Computational pathology approach for retrospective analysis of tissue-based companion diagnostic driven clinical trial studies

ABSTRACT

Automated systems and methods are presented for retrospectively analyzing clinical trial data. A plurality of image derived from biological samples of patients in a cohort population are accessed. Image features are computed based on the plurality of images. A diagnostic feature metric is derived based on the computed image features. A cut point value is determined by applying a statistical minimization method using the derived diagnostic feature metric and patient outcome data from the cohort population, in which the cut point value identifies a patient in the cohort population as positive or negative for a diagnostic test.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of International ApplicationPCT/EP2018/085308, entitled “Computational Pathology Approach ForRetrospective Analysis Of Tissue-Based Companion Diagnostic DrivenClinical Trial Studies” and filed Dec. 17, 2018, which claims priorityto U.S. Provisional Application No.: 62/610,216, filed Dec. 24, 2017.Each of these applications is hereby incorporated by reference in itsentirety for all purposes.

BACKGROUND

In pharmaceutical development, one critical aspect of the processinvolves performing clinical trials of a proposed new pharmaceutical ormedical device in preparation for regulatory approval. Such clinicaltrials involve usage of the proposed new pharmaceutical or medicaldevice on a large number of patients and monitoring of results andpotential side effects of such usage in each patient. Such clinicaltrials are done in phases and span a significant period of time.Extensive and often complex clinical trial protocols are developed thatdefine, for example, targeted demographics, proposed medications,patient regimens, forms for collection, types of statistically relevantdata, the timing or order of events within the study, often even thelayout of the reporting data, or other suitable data.

The clinical trials are performed in a series of phases, known as PhaseI, Phase II, and Phase III. Each phase varies in duration, the number ofpatients involved and purpose. Failure at any stage of Phases I, II orIII of the clinical trial process effectively ends the therapy's chancesfor final approval.

Before entering Phase I, a sponsor needs to obtain regulatory approval.Phase I trials typically last six months and involve tens of volunteersubjects usually all of whom are located at a single investigative site.Phase I trials test the safety of the therapy. Once Phase I trials arecomplete and the therapy has been shown to be safe, the sponsor requestspermission from the regulatory authority to proceed with furtherclinical tests.

Phase II trials typically last six to twelve months, involve tens tohundreds of patients and are conducted to test the effectiveness of thetreatment, device or drug. A sponsor may conduct many Phase II trials,attempting to find as many uses of the therapy as possible. If thetherapy appears to be effective, the sponsor requests permission fromthe regulatory authority to proceed with large scale trials.

For each likely use of the therapy, the sponsor conducts at least twoPhase III trials. Phase III trials typically last 24 to 36 months andinvolve thousands of patients. Phase III trials are blinded trials, thatis, a portion of the patients receive the therapy and the remainingpatients receive a placebo or active control, and the identities ofpatients taking the trial therapy are not known to anyone until thetrial is complete. Phase III trials are conducted to test the safety andeffectiveness of a therapy in a large population. The Phase III trial isthe first opportunity to observe infrequent adverse effects in thegeneral population; each and every one is carefully recorded. Since theeffectiveness of the therapy is tested in a blinded environment, theresults are not known until after the study is complete.

A good number of clinical trials that succeed in Phase II fail in PhaseIII, and sometimes the reason for the failure is unknown or not fullyappreciated. For example, it is believed that about 40% of cancer drugsfail in Phase III. When a clinical study that is successful in Phase IIis unsuccessful in Phase III, possible reasons for failure are include(i) the Phase II data was not representative enough of the broaderpatient pool in Phase III; (ii) the underlying target biology and otherinteractions were not well understood; and (iii) the wrong patients wereenrolled.

In some clinical studies, a companion diagnostic is used to select theintended patients. Companion diagnostic evaluation in done by analyzingpatient tissue (which can be evaluating protein expression in tissueslides, or molecular or genomic analysis of the patient tissue, etc.)and a threshold may be selected for inclusion of the specific patientinto the trial. Companion diagnostics should provide reproducibleresults.

SUMMARY

It is often necessary to understand the outcomes of clinical trial data.For example, and as noted above, if a Phase II clinical trial issuccessful but a subsequent Phase III clinical trial is not, it would beadvantageous to understand why the Phase III trial was not successful.The systems and methods described herein enable the skilled artisan toretrospectively analyze clinical trial data, i.e. patient outcome dataand/or collected image data corresponding to patient biological samples,and decipher unexpected clinical trial outcomes, or guide medicalprofessionals in identifying necessary changes before further clinicaltrials are conducted.

Within this in mind, in one aspect of the present disclosure is anautomated method for deriving a diagnostic cut point, the diagnostic cutpoint used to identify a patient in a cohort population as positive ornegative for a particular diagnostic test comprising: (a) computing oneor more image feature metrics from a plurality of images derived frombiological samples of patients in the cohort population, the biologicalsamples having at least one stain; (b) deriving a diagnostic featuremetric based on the computed image feature metrics; and (c) applying astatistical minimization to derive the cut point value, the statisticalminimization utilizing the derived diagnostic feature metric and patientoutcome data from the cohort population. In some embodiments, thepatient cohort is a Phase II and/or Phase III patient cohort. In someembodiments, the patient cohort is a Phase II placebo cohort, a PhaseIII placebo cohort, or both a Phase II and Phase III placebo cohort(i.e. cohorts that did not receive an experimental drug or treatmentprotocol). In some embodiments, the patient outcome data is a clinicalendpoint. In some embodiments, the clinical endpoint is primary endpoint data. In some embodiments, the primary end point data is at leastone of overall patient survival time, recurrence free survival, diseasefree survival, drug response, response duration, progression freesurvival, or pathological complete response. In some embodiments, theclinical endpoint is secondary endpoint data. In some embodiments, thepatient outcome data for each cohort population is stored in a database.

In some embodiments, slides from the biological samples are digitized,and an image analysis algorithm is used to derive one or more imagefeature metrics. In some embodiments, the image analysis algorithmdetects and classifies cells and/or nuclei within the input images,whereby the classification results may be utilized to generate one ormore expression scores. In some embodiments, the generated expressionscores may be utilized as diagnostic feature metrics. In someembodiments, the expression score is an H-score. In some embodiments,the expression score is biomarker percent positivity. In someembodiments, the expressions core is an Allred score.

In some embodiments, the diagnostic feature metric is a combination ofmultiple image feature metrics or expression scores. In someembodiments, the multiple image feature metrics or expression scores arecombined using a proportional hazard model. In some embodiments, theproportional hazard model is a multivariate Cox model. In someembodiments, the multiple image feature metrics or expression scoresthat are combined with the multivariate Cox model are pre-determined(e.g. determined by a pathologist or other medical profession, or basedon diagnostic guidelines).

In some embodiments, the diagnostic feature metric is a combination ofmultiple image feature metrics or expression scores which are determinedbased on machine learning, i.e. using a classifier trained to determinethose image feature metrics that best stratify patients when presentedwith patient outcome data and image analysis data. In some embodiments,the multiple image feature metrics or expression scores that aredetermined through machine learning are combined in a multivariate Coxmodel. In some embodiments, a classifier for machine learning is builtusing image data and patient outcome data from of a placebo cohort, atreatment arm cohort, or both the placebo and treatment arm cohorts. Insome embodiments, the trained classifier will determine a top N numberof image feature metrics out of a total M number of image featuremetrics that best stratify patients based on patient outcome datapresented to the classifier.

In some embodiments, the statistical minimization method is a log rankstatistic minimization. In some embodiments, the method furthercomprises stratifying the patients into diagnostic positive anddiagnostic negative groups based on the determined cut point value. Insome embodiments, the method further comprises generating Kaplan-Meierresponse curves. In some embodiments, the method further comprisescalculating hazard ratios based on the generated Kaplan-Meier responsecurves. In some embodiments, the method further comprises comparing thedetermined cut point value to a manually selected diagnostic cutoffvalue.

In some embodiments, the images are of biological samples stained forthe detection of breast cancer biomarkers. In some embodiments, theimages are of biological samples stained for the detection of non-smalllung cell cancer biomarkers.

In another aspect of the present disclosure is a system whichretrospectively analyzes clinical trial data, the system comprising: (i)one or more processors, and (iii) a memory coupled to the one or moreprocessors, the memory to store computer-executable instructions that,when executed by the one or more processors, cause the one or moreprocessors to perform operations comprising: (a) computing one or moreimage feature metrics from a plurality of images derived from biologicalsamples of patients in a cohort population, the biological sampleshaving at least one stain; (b) deriving a diagnostic feature metricbased on the computed image feature metrics, and (c) applying astatistical minimization method to derive a cut point value, wherein thestatistical minimization method takes into account the deriveddiagnostic feature metric and patient outcome data from the cohortpopulation. In some embodiments, the patient cohort is a Phase II and/orPhase III patient cohort. In some embodiments, the patient cohort is aPhase II placebo cohort, a Phase III placebo cohort, or both a Phase IIand Phase III placebo cohort (i.e. cohorts that did not receive anexperimental drug or treatment protocol). In some embodiments, thepatient outcome data is a primary end point data. In some embodiments,the primary end point data is at least one of overall patient survivaltime, recurrence free survival, drug response, or pathological completeresponse. In some embodiments, the statistical minimization method is alog rank statistic minimization.

In some embodiments, the diagnostic feature metric is derived throughmultivariate Cox modeling taking into account multiple computed imagefeature metrics. In some embodiments, the multiple computed imagefeature metrics for Cox modeling are predetermined (e.g. multipleexpression scores as determined by a pathologist).

In some embodiments, machine learning is used to determine thosecomputed image feature metrics that most accurately may be used tostratify patient cohorts. In some embodiments, the image feature metricsthat most accurately may be used to stratify patient cohorts are fed toa multivariate Cox model to provide the diagnostic feature metric.

In some embodiments, the system further comprises instructions forstratifying the patients into diagnostic positive and diagnosticnegative groups based on the determined cut point value. In someembodiments, the system further comprises instructions for generatingKaplan-Meier response curves. In some embodiments, the system furthercomprises instructions for calculating hazard ratios based on thegenerated response curves. In some embodiments, the system furthercomprises instructions for comparing the determined cut point value to amanually selected diagnostic cutoff value.

In another aspect of the present disclosure is a non-transitorycomputer-readable medium including instructions for retrospectivelyanalyzing clinical trial data comprising: (a) computing one or moreimage feature metrics from a plurality of images derived from biologicalsamples of patients in a cohort population, the biological sampleshaving at least one stain; (b) deriving a diagnostic feature metricbased on the computed image feature metrics, and (c) applying astatistical minimization method to derive a cut point value, wherein thestatistical minimization method takes into account the deriveddiagnostic feature metric and patient outcome data from the cohortpopulation. In some embodiments, the patient cohort is a Phase II and/orPhase III patient cohort. In some embodiments, the patient cohort is aPhase II placebo cohort, a Phase III placebo cohort, or both a Phase IIand Phase III placebo cohort (i.e. cohorts that did not receive anexperimental drug or treatment protocol). In some embodiments, thepatient outcome data is a primary end point data. In some embodiments,the primary end point data is at least one of overall patient survivaltime, recurrence free survival, drug response, or pathological completeresponse. In some embodiments, the statistical minimization method is alog rank statistic minimization.

In some embodiments, the diagnostic feature metric is derived frommultivariate Cox modeling. In some embodiments, the multivariate Coxmodel is built using multiple computed image feature metrics. In someembodiments, the multiple computed image feature metrics arepredetermined. In some embodiments, the multiple computed image featuremetrics are determined through machine learning.

In some embodiments, the multivariate Cox model is built from imagefeature metrics determined by machine learning to best stratify patientcohorts. In some embodiments, patient output data and image analysisdata are supplied to a classifier such that those image feature metricsthat best correlate with patient outcome data may be determined.

In another aspect of the present disclosure is a method of comparingsets of data from two patient populations comprising: (a) computing oneor more image feature metrics from a plurality of images of biologicalsamples derived from a first patient population; (b) combining thecomputed one or more image feature metrics from the first patientpopulation with tissue analysis data corresponding to the biologicalsamples derived from the first patient population to provide tissuefeature data for the first patient population; (c) computing a firstpatient population correlation matrix based (i) on the tissue featuredata for the first patient population, and (ii) clinical attributes ofthe first patient population; (d) computing one or more image featuremetrics from a plurality of images of biological samples derived from asecond patient population; (e) combining the computed one or more imagefeature metrics from the second patient population with tissue analysisdata corresponding to the biological samples derived from the secondpatient population to provide tissue feature data for the second patientpopulation; (f) computing a second patient population correlation matrixbased (i) on the tissue feature data for the second patient population,and (ii) clinical attributes of the second patient population; and (g)determining whether the tissue feature data for the first patientpopulation is similar to the tissue feature data for the second patientpopulation by comparing the first and second patient populationcorrelation matrices.

In some embodiments, the tissue analysis data includes molecular and/orgenomic features. In some embodiments, the clinical attributes areselected from the group consisting of age, weight, immune response, sex,and ethnicity. In some embodiments, at least 50 tissue features arecollected for each patient population.

In some embodiments, the first patient population is a Phase II patientcohort, while the second patient population is a Phase III patientcohort. In some embodiments, the first patient population is a Phase IIplacebo cohort, while the second patient population is a Phase IIIplacebo cohort. In some embodiments, the first patient population is aPhase II test arm cohort, while the second patient population is a PhaseIII placebo cohort. In some embodiments, the first patient population isa Phase II test arm cohort, while the second patient population is aPhase III test arm cohort. In some embodiments, the first patientpopulation is a Phase II placebo cohort, while the second patientpopulation is a Phase II test arm cohort. In some embodiments, the firstpatient population is data collected pre-treatment; while the secondpatient population is data collected from the same patientspost-treatment.

In another aspect of the present disclosure is a method of comparingsets of data from two patient populations comprising: (a) computing oneor more image feature metrics from a plurality of images of biologicalsamples derived from a first patient population; (b) combining thecomputed one or more image feature metrics from the first patientpopulation with tissue analysis data corresponding to the biologicalsamples derived from the first patient population to provide tissuefeature data for the first patient population; (c) computingdistributions of individual tissue features from the first patientpopulation; (d) computing one or more image feature metrics from aplurality of images of biological samples derived from a second patientpopulation; (e) combining the computed one or more image feature metricsfrom the second patient population with tissue analysis datacorresponding to the biological samples derived from the second patientpopulation to provide tissue feature data for the second patientpopulation; (f) computing distributions of individual tissue featuresfor the second patient population; and (g) determining whether thetissue feature data for the first patient population is similar to thetissue feature data for the second patient population by comparing thecomputed distributions of individual tissue features from the first andsecond patient populations.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a representative digital pathology system includingan image acquisition device and a computer system.

FIG. 2 sets forth various modules that can be utilized in a digitalpathology system or within a digital pathology workflow.

FIG. 3 sets forth a flowchart illustrating the steps of deriving a cutpoint and using the derived cut point in generating drug response curvesand/or analyzing clinical trial data.

FIG. 4 provides a flow chart illustrating the steps of region selection.

FIG. 5 sets forth a flowchart illustrating the steps of applying machinelearning to retrospectively analyze clinical trial data.

FIG. 6 provides an example of a drug-response curve.

FIG. 7 sets forth a flow chart illustrating the steps of deriving cohortsignatures and comparing how similar or how different patientpopulations are to each other.

DETAILED DESCRIPTION

It should also be understood that, unless clearly indicated to thecontrary, in any methods claimed herein that include more than one stepor act, the order of the steps or acts of the method is not necessarilylimited to the order in which the steps or acts of the method arerecited.

As used herein, the singular terms “a,” “an,” and “the” include pluralreferents unless context clearly indicates otherwise. Similarly, theword “or” is intended to include “and” unless the context clearlyindicates otherwise. The term “includes” is defined inclusively, suchthat “includes A or B” means including A, B, or A and B.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of or “exactly one of,” or, when used inthe claims, “consisting of,” will refer to the inclusion of exactly oneelement of a number or list of elements. In general, the term “or” asused herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of” or“exactly one of.” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

As used herein, the terms “comprising,” “including,” “having,” and thelike are used interchangeably and have the same meaning. Similarly,“comprises,” “includes,” “has,” and the like are used interchangeablyand have the same meaning. Specifically, each of the terms is definedconsistent with the common United States patent law definition of“comprising” and is therefore interpreted to be an open term meaning “atleast the following,” and is also interpreted not to exclude additionalfeatures, limitations, aspects, etc. Thus, for example, “a device havingcomponents a, b, and c” means that the device includes at leastcomponents a, b and c. Similarly, the phrase: “a method involving stepsa, b, and c” means that the method includes at least steps a, b, and c.Moreover, while the steps and processes may be outlined herein in aparticular order, the skilled artisan will recognize that the orderingsteps and processes may vary. The term “amplification,” as used herein,refers to a process of multiplying an original quantity of a nucleicacid template in order to obtain greater quantities of the originalnucleic acid.

As used herein, the term “biological sample” or “tissue sample” refersto any sample including a biomolecule (such as a protein, a peptide, anucleic acid, a lipid, a carbohydrate, or a combination thereof) that isobtained from any organism including viruses. Other examples oforganisms include mammals (such as humans; veterinary animals like cats,dogs, horses, cattle, and swine; and laboratory animals like mice, ratsand primates), insects, annelids, arachnids, marsupials, reptiles,amphibians, bacteria, and fungi. Biological samples include tissuesamples (such as tissue sections and needle biopsies of tissue), cellsamples (such as cytological smears such as Pap smears or blood smearsor samples of cells obtained by microdissection), or cell fractions,fragments or organelles (such as obtained by lysing cells and separatingtheir components by centrifugation or otherwise). Other examples ofbiological samples include blood, serum, urine, semen, fecal matter,cerebrospinal fluid, interstitial fluid, mucous, tears, sweat, pus,biopsied tissue (for example, obtained by a surgical biopsy or a needlebiopsy), nipple aspirates, cerumen, milk, vaginal fluid, saliva, swabs(such as buccal swabs), or any material containing biomolecules that isderived from a first biological sample. In certain embodiments, the term“biological sample” as used herein refers to a sample (such as ahomogenized or liquefied sample) prepared from a tumor or a portionthereof obtained from a subject.

As used herein, the terms “biomarker” or “marker” refer to a measurableindicator of some biological state or condition. In particular, abiomarker may be a protein or peptide, e.g. a surface protein, that canbe specifically stained and which is indicative of a biological featureof the cell, e.g. the cell type or the physiological state of the cell.An immune cell marker is a biomarker that is selectively indicative of afeature that relates to an immune response of a mammal. A biomarker maybe used to determine how well the body responds to a treatment for adisease or condition or if the subject is predisposed to a disease orcondition. In the context of cancer, a biomarker refers to a biologicalsubstance that is indicative of the presence of cancer in the body. Abiomarker may be a molecule secreted by a tumor or a specific responseof the body to the presence of cancer. Genetic, epigenetic, proteomic,glycomic, and imaging biomarkers can be used for cancer diagnosis,prognosis, and epidemiology. Such biomarkers can be assayed innon-invasively collected biofluids like blood or serum. Several gene andprotein based biomarkers have already been used in patient careincluding but, not limited to, AFP (Liver Cancer), BCR-ABL (ChronicMyeloid Leukemia), BRCA1/BRCA2 (Breast/Ovarian Cancer), BRAF V600E(Melanoma/Colorectal Cancer), CA-125 (Ovarian Cancer), CA19.9(Pancreatic Cancer), CEA (Colorectal Cancer), EGFR (Non-small-cell lungcarcinoma), HER-2 (Breast Cancer), KIT (Gastrointestinal stromal tumor),PSA (Prostate Specific Antigen), S100 (Melanoma), and many others.Biomarkers may be useful as diagnostics (to identify early stagecancers) and/or prognostics (to forecast how aggressive a cancer isand/or predict how a subject will respond to a particular treatmentand/or how likely a cancer is to recur).

As used herein, the phrase “Cox proportional hazard model” refers to amodel which is expressed mathematically as h(x,t)=h₀(t)×exp{b₁x₁+b₂x₂+ .. . +b_(p)x_(p)} wherein h(x,t) is the expected hazard at time t and b₁,b₂ . . . b_(p) are constants extrapolated for each of the independentvariables.

As used herein, the terms “cutoff” or “clinical cutoff” refer, in thecontext of treatment with a therapeutic product, a value set by taking arisk-benefit balance of the therapeutic product into account. Forexample, and in the context of a companion diagnostic measuring thepresence or absence of a particular biomarker, subjects are consideredto be biomarker-positive if they are above a predetermined cutoff value;and biomarker-negative if they are below a predetermined cutoff value.Treatment decisions will be made with respect to the groups divided bysuch a clinical cut-off

As used herein, the term “companion diagnostic” refers to a medicaldevice or assay which provides information that is essential for thesafe and effective use of a corresponding drug or biological product.The test helps a health care professional determine whether a particulartherapeutic product's benefits to patients will outweigh any potentialserious side effects or risks. In some embodiments, the clinicalperformance of the companion diagnostic is the ability of the testdeveloped for a predictive biomarker (the companion diagnostic) todistinguish treatment responders from non-responders. Companiondiagnostics can: (i) identify patients who are most likely to benefitfrom a particular therapeutic product; (ii) identify patients likely tobe at increased risk for serious side effects as a result of treatmentwith a particular therapeutic product; and/or (iii) monitor response totreatment with a particular therapeutic product for the purpose ofadjusting treatment to achieve improved safety or effectiveness. Theclinical performance of the companion diagnostic not only directlyaffects the number of patients who are potentially eligible fortreatment but also affects the net benefit enrichment achieved, aspatients who are selected by the companion diagnostic and arenon-responders also receive treatment, thereby reducing the observedaverage response. As such, if the diagnostic test is inaccurate, thenthe treatment decision based on that test may not be optimal.

As used herein, the terms “endpoints” or “outcomes” describe and definethe goal(s) of a clinical study. Examples of endpoints (which varydepending on the type and phase of trial) include overall survival,toxicity, tumor response, patient survival or quality of life.

As used herein, the term “image data” encompasses raw image dataacquired from the biological tissue sample, such as by means of anoptical sensor or sensor array, or pre-processed image data. Inparticular, the image data may comprise a pixel matrix.

As used herein, the term “immunohistochemistry” refers to a method ofdetermining the presence or distribution of an antigen in a sample bydetecting interaction of the antigen with a specific binding agent, suchas an antibody. A sample is contacted with an antibody under conditionspermitting antibody-antigen binding. Antibody-antigen binding can bedetected by means of a detectable label conjugated to the antibody(direct detection) or by means of a detectable label conjugated to asecondary antibody, which binds specifically to the primary antibody(indirect detection).

As used herein, the terms “placebo” or “control” refer to a group ofpatients that receive an inactive substance while the drug beingevaluated is given to another group: designed to compare efficacy of thedrug with ‘no’ treatment. Placebo-controlled trials are rarely used forcancer treatments, where a new treatment is more likely to be comparedwith the existing standard-of-care treatment.

As used herein, the term “stratification” refers to a way of groupingsubsets of patients and is used in randomized trials when factors thatcan influence the intervention's success are known. For example,participants whose cancer has spread from the original tumor site can beseparated, or stratified, from those whose cancer has not spread, sinceit might be expected that these patients have more advanced and lessadvanced disease respectively and could respond differently to treatmentinterventions.

Overview

The present disclosure provides automated systems and methods forretrospectively analyzing clinical trial data to disambiguate and betterunderstand why a clinical trial was successful or unsuccessful. Indeed,utilization of the systems and methods described herein allows for theskilled artisan to better understand why, for instance, a successfulPhase II study was not as successful during Phase III trials or why thePhase III trial outright failed. For example, by using the systems andmethods described herein, the skilled artisan will be able to appreciatewhether the Phase II data was representative enough of the broaderpatient pool introducing during Phase III trials. Likewise, the skilledartisan will be able to make a determination of whether the appropriatepatients were enrolled within the clinical study, based on the choice ofthe tissue-based companion diagnostic, or whether the thresholds for thecompanion-diagnostic were appropriately selected.

In view of this, the present disclosure provides, in some embodiments,methods for determining an optimal cut point based on image feature dataand patient outcome data. The determination of the optimal cut point mayallow for the generation of drug response curves, such that generateddrug response curves may yield hazard ratios. Likewise, thedetermination of the optimal cut point may allow for the comparison withthe predictive cut point for the clinical trial and/or a comparison ofpatient stratification based on the optimal and predictive cut points.These and other features of the present disclosure will be describedherein.

Digital Pathology Systems

A digital pathology system 200 for imaging and analyzing specimens isillustrated in FIGS. 1 and 2. The digital pathology system 200 maycomprise an imaging apparatus 12 (e.g. an apparatus having means forscanning a specimen-bearing microscope slide) and a computer system 14,whereby the imaging apparatus 12 and computer may be communicativelycoupled together (e.g. directly, or indirectly over a network 20). Thecomputer system 14 can include a desktop computer, a laptop computer, atablet, or the like, digital electronic circuitry, firmware, hardware,memory 201, a computer storage medium (240), a computer program or setof instructions (e.g. where the program is stored within the memory orstorage medium), one or more processors (209) (including a programmedprocessor), and any other hardware, software, or firmware modules orcombinations thereof (such as described further herein). For example,the computing system 14 illustrated in FIG. 1 may comprise a computerwith a display device 16 and an enclosure 18. The computer system canstore digital images in binary form (locally, such as in a memory, on aserver, or another network connected device). The digital images canalso be divided into a matrix of pixels. The pixels can include adigital value of one or more bits, defined by the bit depth. The skilledartisan will appreciate that other computer devices or systems may beutilized and that the computer systems described herein may becommunicatively coupled to additional components, e.g. specimenanalyzers, microscopes, other imaging systems, automated slidepreparation equipment, etc. Some of these additional components and thevarious computers, networks, etc. that may be utilized are describedfurther herein.

In general, the imaging apparatus 12 (or other image source includingpre-scanned images stored in a memory) can include, without limitation,one or more image capture devices. Image capture devices can include,without limitation, a camera (e.g., an analog camera, a digital camera,etc.), optics (e.g., one or more lenses, sensor focus lens groups,microscope objectives, etc.), imaging sensors (e.g., a charge-coupleddevice (CCD), a complimentary metal-oxide semiconductor (CMOS) imagesensor, or the like), photographic film, or the like. In digitalembodiments, the image capture device can include a plurality of lensesthat cooperate to prove on-the-fly focusing. An image sensor, forexample, a CCD sensor can capture a digital image of the specimen. Insome embodiments, the imaging apparatus 12 is a brightfield imagingsystem, a multispectral imaging (MSI) system or a fluorescent microscopysystem. The digitized tissue data may be generated, for example, by animage scanning system, such as a VENTANA iScan HT scanner by VENTANAMEDICAL SYSTEMS, Inc. (Tucson, Ariz.) or other suitable imagingequipment. Additional imaging devices and systems are described furtherherein. The skilled artisan will appreciate that the digital color imageacquired by the imaging apparatus 12 is conventionally composed ofelementary color pixels. Each colored pixel can be coded over threedigital components, each comprising the same number of bits, eachcomponent corresponding to a primary color, generally red, green orblue, also denoted by the term “RGB” components.

FIG. 2 provides an overview of the various modules utilized within thepresently disclosed digital pathology system. In some embodiments, thedigital pathology system 200 employs a computer device orcomputer-implemented method having one or more processors 203 and atleast one memory 201, the at least one memory 201 storing non-transitorycomputer-readable instructions for execution by the one or moreprocessors to cause the one or more processors to execute instructions(or stored data) in one or more modules (e.g. modules 202, and 205through 211). Alternatively, the instructions may be stored in anon-transitory computer-readable medium (201) or computer-usable medium.

With reference to FIGS. 2 and 3, the present disclosure provides asystem and method of retrospectively analyzing clinical trial data, thesystem and method comprising (a) an image acquisition module 202 togenerate or receive simplex or multiplex images, e.g. acquired images ofa biological sample stained with one or more stains (step 300); (b)running feature extraction module 205 to derive image feature metricsfrom the input images (step 310); (c) running an optional classificationmodule 206 to classify cells and/or nucleic within the input images (d)running an optional scoring module 207 to score image data using thederived image feature metrics and/or classification results; (e) runningan optional multivariate Cox model module 208 to derive a diagnosticfeature based on a weighted combination of multiple image featuremetrics or expressions; (f) running an optional prognostic featurederivation module 209 to determine the most relevant features coincidingwithin patient outcome data; (g) running a cutoff determination module210 to determine a cut point (step 330), which facilitates thestratification of patients into diagnostic positive and negative groups(step 330) and enables statistical comparisons to be made (steps 350 and360); and (h) running an optional drug response curve generation module211 to compute drug response curves for different patient populations(step 370). The skilled artisan will also appreciate that additionalmodules may be incorporated into the workflow. As will be described inmore detail here, in some embodiments, an image processing module may berun to apply certain filters to the acquired images or to identifycertain histological and/or morphological structures or features withinthe tissue samples. Likewise, a region of interest selection module maybe utilized to select a particular portion of an image for analysis.

FIG. 3 sets forth a flowchart which provides a general overview of themethods of the presently disclosed workflow. In general, the methodincludes receiving image data from a first patient population (step300); deriving a diagnostic feature metrics from the received image data(step 320) by extracting image feature metrics from input images (step310); and determining an optimal cutoff value to identify a patient aspositive or negative for a diagnostic (step 330), the optical cutoffbeing determined using derived diagnostic feature metrics and patientoutcome data (such as outcome data stored in database 212). In someembodiments, the method further comprises the step of stratifyingpatients into diagnostic positive and negative groups (step 340) andcomparing the determined optimal cutoff and predictive stratification tothe manually selected diagnostic cut point and manually selectedstratification (step 350). In some embodiments, the method furthercomprises the step of determining whether the correct companiondiagnostic was used and/or the impact of the optimal cutoff (step 360).In some embodiments, the method further comprises the step of generatingdrug response curves (step 370).

Patient Outcome Database

With reference to FIG. 2, the digital pathology systems of the presentdisclosure may include a patient outcome database 212 which serves as arepository of data pertaining to clinical trials for particular patientcohorts under study. Indeed, the database 212 may facilitate the storageof any primary and secondary endpoint data, as well as any associatedpatient data (e.g. patient name or identification, age, sex, weight,ethnicity, tumor size, tumor type, genetic information, pathologicalfinds, etc.), whereby the data stored therein may be retrieved by thedigital pathology system 200 and be used in further downstreamprocessing (e.g. statistical analyses).

As the skilled artisan will appreciate, the aim of a clinical trial isto measure key outcomes or endpoints and to test the clinical efficacyand tolerability of the treatment in a particular disease. In someembodiments, trial will usually specify a primary endpoint. This is themost important endpoint of the trial and, if met, means a positiveresult for the trial and the treatment. In general, in a clinicalresearch trial, a clinical endpoint generally refers to occurrence of adisease, symptom, sign or laboratory abnormality that constitutes one ofthe target outcomes of the trial. A clinical trial will usually defineor specify a primary endpoint as a measure that will be consideredsuccess of the therapy under trial (e.g. in justifying a marketingapproval). The clinical trial protocol provides the design for the studyconduct and sets out the endpoints of the study up-front. There is clearguidance on how and when to measure and evaluate the study endpoints.

In some embodiments, the primary endpoint might be a statisticallysignificant improvement in overall survival (“OS”), i.e. the time fromrandomization until death from any cause. Overall survival is defined asthe time from randomization until death from any cause and is documentedby the date of death. Overall survival can be measured in two ways:either as median overall survival, which is a duration of time at which50% of patients in the trial are alive, or as a percentage of patientsalive at different time points during the trial, which may be measuredat 1, 2, or 5 years. Median overall survival is often used as a primaryor co-primary endpoint. In some cases, post-marketing studies willcontinue in order to capture overall survival after initial efficacy isvalidated.

In some embodiments, overall survival is reported as a five-yearsurvival rate, i.e. percentage of patients alive five years afterdiagnosis or treatment. The overall survival rates reported afterdiagnosis of different diseases can vary, since some cancers have abetter outlook (survival rate) than others. The effect of a treatment onoverall survival should be viewed relative to the background or expectedoverall survival for a given cancer.

A trial might also define one or more secondary endpoints. Suchsecondary endpoints include, without limitation,progression-free-survival (PFS) (i.e. the time from randomization untildisease progression or death); time to progression (TTP) (i.e. the timefrom randomization until objective tumor progression; does not includedeaths); time to treatment failure (TTF) (i.e. time from randomizationto discontinuation of treatment for any reason, including diseaseprogression, treatment toxicity, and death); and event-free survival(EFS) (i.e. time from randomization to disease progression, death, ordiscontinuation of treatment for any reason (e.g., toxicity, patientpreference, or initiation of a new treatment without documentedprogression)). As a specific example, PFS-6 is the rate, or proportionof patients given a treatment that survive without their diseaseworsening at six months after treatment began.

Response rate (RR) measures tumor size, usually using a scan or X-ray.It gives an indication of whether the tumor is responding to atreatment—if the tumor size has shrunk, it is deemed that there has beena “response”. There are different ways of determining response rate andthe internationally recognized RECIST (Response Evaluation Criteria InSolid Tumors) guidelines are often used in clinical trials.

The trial design is not complete when the trial population, treatmentand endpoints have been identified and defined. In phase III and somephase II trials in cancer, the patient population may be randomized(randomly allocated to receive one or other of the alternativetreatments being studied) and stratified (partitioned by a factor otherthan the treatment, often to ensure that equal numbers of participantswith a characteristic thought to affect prognosis or response to theintervention will be allocated to each comparison group). The goldstandard in clinical research is a scientifically rigorous, randomized,well controlled trial.

Image Acquisition Module

With reference to FIG. 2, the digital pathology system 200 runs an imageacquisition module 202 to capture images or image data of a biologicalsample having one or more stains (i.e. the images may be simplex imagesor multiplex images). In some embodiments, the images captured arestored in memory 201. In some embodiments, the image acquisition module202 is a database or memory comprising previously digitized and storedimages from patient biological samples stained with one or more stains(or a plurality of digital images for each patient in a cohort ofpatients). In some embodiments, the images received or acquired are RGBimages or multispectral images. In some embodiments, in place of thecaptured raw images, any set of optional pre-processed images from thecaptured raw images can be used, either as an independent input imagesor in combination with the captured raw images. Accordingly, similarpre-processing step can be used when applying the trained network to anunlabeled image, as described herein.

In some embodiments, image data is received from a specific patientpopulation undergoing a clinical trial. For example, image data may bereceived for a Phase II cohort and/or a Phase II cohort that receivedplacebo.

The images or image data (used interchangeably herein) may be acquiredusing the imaging apparatus 12, such as in real-time. In someembodiments, the images are acquired from a microscope or otherinstrument capable of capturing image data of a specimen-bearingmicroscope slide, as noted herein. In some embodiments, the images areacquired using a 2D scanner, such as one capable of scanning imagetiles, or a line scanner capable of scanning the image in a line-by-linemanner, such as the VENTANA DP 200 scanner. Alternatively, the imagesmay be images that have been previously acquired (e.g. scanned) andstored in a memory 201 (or, for that matter, retrieved from a server vianetwork 20).

In some embodiments, the images (again, either simplex or multipleximages) received as input are derived from serial tissue sections, i.e.serial sections derived from the same xenograft tissue block. Ingeneral, the at least two images received as input each comprise signalscorresponding to a stain (including chromogens, fluorophores, quantumdots, etc.). In some embodiments, the images have been stained with aleast one primary stain (hematoxylin or eosin) and/or have been stainedin at least one of an IHC assay or ISH assay for the identification of aspecific biomarker (referred to herein as a “biomarker” image). In someembodiments, multiple images are received for each patient in a clinicalstudy, and at least one of the images has been stained with bothhematoxylin and eosin (referred to herein as an “H&E image”), whileanother one of the images has been stained in at least one of an IHCassay or ISH assay for the identification of a specific biomarker. Insome embodiments, the input images may be multiplex images, i.e. stainedfor multiple, different markers in a multiplex assay according tomethods known to those of ordinary skill in the art.

A typical biological sample is processed in a staining/assay platformthat applies a stain to the sample. In some embodiments, specimenprocessing apparatus is an automated apparatus, such as the BENCHMARK XTinstrument, the SYMPHONY instrument, the BENCHMARK ULTRA instrument soldby Ventana Medical Systems, Inc. Ventana Medical Systems, Inc. is theassignee of a number of United States patents disclosing systems andmethods for performing automated analyses, including U.S. Pat. Nos.5,650,327, 5,654,200, 6,296,809, 6,352,861, 6,827,901 and 6,943,029, andU.S. Published Patent Application Nos. 20030211630 and 20040052685, eachof which is incorporated herein by reference in its entirety for allpurposes. Alternatively, specimens can be manually processed.

Examples of commercially available H&E stainers include the VENTANASYMPHONY (individual slide stainer) and VENTANA HE 600 (individual slidestainer) series H&E stainers from Roche; the Dako CoverStainer (batchstainer) from Agilent Technologies; the Leica ST4020 Small LinearStainer (batch stainer), Leica ST5020 Multistainer (batch stainer), andthe Leica ST5010 Autostainer XL series (batch stainer) H&E stainers fromLeica Biosystems Nussloch GmbH. Other commercial products on the marketsuitable for use as the staining/assay platform, one example being theDiscovery™ product of Ventana Medical Systems, Inc. (Tucson, Ariz.).

The camera platform may also include a bright field microscope, oneexample being the VENTANA iScan HT product of Ventana Medical Systems,Inc., or any microscope having one or more objective lenses and adigital imager, as well as a set of spectral filters. Other techniquesfor capturing images at different wavelengths may be used. Furthercamera platforms suitable for imaging stained biological specimens areknown in the art and commercially available from companies such asZeiss, Canon, Applied Spectral Imaging, and others, and such platformsare readily adaptable for use in the system, methods and apparatus ofthis subject disclosure.

As the skilled artisan will appreciate, a tissue sample may be stainedfor different types of nuclei and/or cell membrane biomarkers. Methodsfor staining tissue structures and guidance in the choice of stainsappropriate for various purposes are discussed, for example, in“Sambrook et al., Molecular Cloning: A Laboratory Manual, Cold SpringHarbor Laboratory Press (1989)” and “Ausubel et al., Current Protocolsin Molecular Biology, Greene Publishing Associates andWiley-Intersciences (1987),” the disclosures of which are incorporatedherein by reference in its entirety for all purposes.

By way of one non-limiting example, and in the context of detectingbreast cancer, in some embodiments the tissue sample is stained in anIHC assay for the presence of one or biomarkers including an estrogenreceptor marker, a progesterone receptor marker, a Ki-67 marker, or aHER2 marker. As such, in some embodiments, the biomarker image used asan input is an IHC image which comprises signals (the signalscorresponding to stains which may be, for example, chromogenic orfluorescent) corresponding to a presence of at least one of an estrogenreceptor (ER) marker, a progesterone receptor (PR) marker, a Ki-67marker, or a HER2 marker. In some embodiments, the sample can beanalyzed to detect or measure the presence of ER, HER2, Ki-67 and PRproteins in the sample, for example a qualitative or quantitativemeasurement. In some embodiments, the expression patterns of ER, HER2,Ki-67 and PR proteins can also be used to determine the heterogeneity ofthe protein expression, such as between different tumor or cell clustersas described further herein. In some examples, the antibodies for ER,PR, HER2 and Ki-67 are obtained from Ventana Medical Systems, Inc.(Tucson, Ariz.). However, one skilled in the art will appreciate thatother antibodies that can be used in the methods and kits providedherein are commercially available from other sources, such as: NovusBiologicals (Littleton, Colo.), Santa Cruz biotechnology, Inc. (SantaCruz, Calif.), Abeam (Cambridge, Mass.), and Invitrogen (Carlsbad,Calif.).

By way of another non-limiting example, and in the context of detectingnon-small cell lung cancer, in some embodiments the tissue sample isstained in an IHC assay for the presence of one or biomarkers includinga PD-L1 biomarker. As such, in some embodiments, the biomarker imageused as an input is an IHC image which comprises signals correspondingto a presence of a PD-L1 marker, CD3 marker and CD8 marker.

In some embodiments, the input images are optionally masked with atissue masking module as described herein. In some embodiments, theinput images are masked such that only tissue regions are present in theimages. In some embodiments, a tissue region mask is generated to masknon-tissue regions from tissue regions. In some embodiments, a tissueregion mask may be created by identifying the tissue regions andexcluding the background regions (e.g. regions of a whole slide imagecorresponding to glass with no sample, such as where there exists onlywhite light from the imaging source). The skilled artisan willappreciate that in addition to masking non-tissue regions from tissueregions, the tissue masking module may also mask other areas of interestas needed, such as a portion of a tissue identified as belonging to acertain tissue type or belonging to a suspected tumor region. In someembodiments, a segmentation technique is used to generate the tissueregion masked images by masking tissue regions from non-tissue regionsin the input images. Suitable segmentation techniques are as such knownfrom the prior art, (cf. Digital Image Processing, Third Edition, RafaelC. Gonzalez, Richard E. Woods, chapter 10, page 689 and Handbook ofMedical Imaging, Processing and Analysis, Isaac N. Bankman AcademicPress, 2000, chapter 2). In some embodiments, an image segmentationtechnique is utilized to distinguish between the digitized tissue dataand the slide in the image, the tissue corresponding to the foregroundand the slide corresponding to the background. In some embodiments, thecomponent computes the Area of Interest (AoI) in a whole slide image inorder to detect all tissue regions in the AoI while limiting the amountof background non-tissue area that is analyzed. A wide range of imagesegmentation techniques (e.g., HSV color-based image segmentation, Labimage segmentation, mean-shift color image segmentation, region growing,level set methods, fast marching methods, etc.) can be used todetermine, for example, boundaries of the tissue data and non-tissue orbackground data. Based at least in part on the segmentation, thecomponent can also generate a tissue foreground mask that can be used toidentify those portions of the digitized slide data that correspond tothe tissue data. Alternatively, the component can generate a backgroundmask used to identify those portions of the digitized slide date that donot correspond to the tissue data.

This identification may be enabled by image analysis operations such asedge detection, etc. A tissue region mask may be used to remove thenon-tissue background noise in the image, for example the non-tissueregions. In some embodiments, the generation of the tissue region maskcomprises one or more of the following operations (but not limited tothe following operations): computing the luminance of the low resolutioninput image, producing a luminance image, applying a standard deviationfilter to the luminance image, producing a filtered luminance image, andapplying a threshold to filtered luminance image, such that pixels witha luminance above a given threshold are set to one, and pixels below thethreshold are set to zero, producing the tissue region mask. Additionalinformation and examples relating to the generation of tissue regionmasks is disclosed in PCT/EP/2015/062015, entitled “An Image ProcessingMethod and System for Analyzing a Multi-Channel Image Obtained from aBiological Tissue Sample Being Stained by Multiple Stains,” thedisclosure of which is hereby incorporated by reference herein in itsentirety for all purposes.

In some embodiments, a region of interest identification module may beused to select a portion of the biological sample for which an image orfor which image data should be acquired. FIG. 4 provides a flow chartillustrating the steps of region selection. In step 420, the regionselection module receives an identified region of interest or field ofview. In some embodiments, the region of interest is identified by auser of a system of the present disclosure, or another systemcommunicatively coupled to a system of the present disclosure.Alternatively, and in other embodiments, the region selection moduleretrieves a location or identification of a region or interest from astorage/memory. In some embodiments, as shown in step 430, the regionselection module automatically generates a FOV or ROI, for example, viamethods described in PCT/EP2015/062015, the disclosure of which ishereby incorporated by reference herein in its entirety for allpurposes. In some embodiments, the region of interest is automaticallydetermined by the system based on some predetermined criteria orcharacteristics that are in or of the image (e.g. for a biologicalsample stained with more than two stains, identifying an area of theimage that comprises just two stains). In step 440, the region selectionmodule outputs the ROI.

Image Analysis

Following the receipt of image data from a patient population (e.g. aPhase II cohort receiving a placebo) or from multiple patientpopulations (e.g. Phase II and Phase III cohorts receiving a placebo),the image data is analyzed. Automated image analysis is utilized suchthat a diagnostic feature metric, derived at least from computed imagefeature metrics, may be provided for downstream processing. In someembodiments, the diagnostic feature metric is selected from one of (i)an expression score; (ii) a Cox hazard ratio derived from multiple,pre-selected image feature metrics and/or expression scores; and (iii) aCox hazard ratio derived from image feature metrics determined to bemost relevant through machine learning techniques.

In some embodiments, a feature extraction module 205 is utilized toderive certain metrics from the received input images. In someembodiments, the derived metrics may be utilized by a classificationmodule 206 such that cells, membranes, and/or nuclei may be identifiedand/or classified, e.g. as being a tumor cell or a non-tumor cell. Insome embodiments, a scoring module 207 may be utilized to score an image(e.g. to provide an expression score), or a portion thereof (e.g. aregion-of-interest), using the derived metrics and/or the classificationresults. In some embodiments, a multivariate Cox model module 208 mayuse a plurality of derived image feature metrics computed using theimage feature extraction module 205 or may use prognostic featuresdetermined to be most relevant through machine learning using theprognostic feature derivation module 209. Each of these modules aredescribed in more detail herein.

(1) Feature Extraction Module

The image data from each patient is first provided to a featureextraction module 205 such that image features may be ascertained (step310). The skilled artisan will appreciate that the nucleus, cytoplasmand membrane of a cell have different characteristics and thatdifferently stained tissue samples may reveal different biologicalfeatures. Indeed, the skilled artisan will appreciate that certain cellsurface receptors can have staining patterns localized to the membrane,or localized to the cytoplasm. Thus, a “membrane” staining pattern isanalytically distinct from a “cytoplasmic” staining pattern. Likewise, a“cytoplasmic” staining pattern and a “nuclear” staining pattern areanalytically distinct. Each of these distinct staining patterns may beused as features for identifying cells and/or nuclei. For example,stromal cells may be strongly stained by FAP, whereas tumor epithelialcells may be strongly stained by EpCAM, while cytokeratins may bestained by panCK. Thus, by utilizing different stains different celltypes may be differentiated and distinguished during image analysis toprovide a classification solution.

Methods of identifying, classifying, and/or scoring nuclei, cellmembranes, and cell cytoplasm in images of biological samples having oneor more stains are described in U.S. Pat. No. 7,760,927 (“the '927Patent”), the disclosure of which is hereby incorporated by referenceherein in its entirety for all purposes. For example, U.S. Pat. No.7,760,927 describes an automated method for simultaneously identifying aplurality of pixels in an input image of a biological tissue stainedwith a biomarker, including considering a first color plane of aplurality of pixels in a foreground of the input image for simultaneousidentification of cell cytoplasm and cell membrane pixels, wherein theinput image has been processed to remove background portions of theinput image and to remove counterstained components of the input image;determining a threshold level between cell cytoplasm and cell membranepixels in the foreground of the digital image; and determiningsimultaneously with a selected pixel and its eight neighbors from theforeground if the selected pixel is cell cytoplasm pixel, a cellmembrane pixel or a transitional pixel in the digital image using thedetermined threshold level.

In some embodiments, tumor nuclei are automatically identified by firstidentifying candidate nuclei and then automatically distinguishingbetween tumor nuclei and non-tumor nuclei. Numerous methods ofidentifying candidate nuclei in images of tissue are known in the art.For example, automatic candidate nucleus detection can be performed byapplying a radial symmetry-based method, such as on the Hematoxylinimage channel or a biomarker image channel after unmixing (see Parvin,Bahram, et al. “Iterative voting for inference of structural saliencyand characterization of subcellular events.” Image Processing, IEEETransactions on 16.3 (2007): 615-623, the disclosure of which isincorporated by reference in its entirety herein for all purposes).

More specifically, in some embodiments the images received as input areprocessed such as to detect nucleus centers (seeds) and/or to segmentthe nuclei. For example, instructions may be provided to detect nucleuscenters based on radial-symmetry voting using the techniques of Parvin(noted above). In some embodiments, nuclei are detected using radialsymmetry to detect centers of nuclei and then the nuclei are classifiedbased on the intensity of stains around the cell centers. In someembodiments, a radial symmetry based nuclei detection operation is usedas described in commonly-assigned and co-pending patent applicationWO/2014/140085A1, the entirety of which is incorporated herein byreference for all purposes. For example, an image magnitude may becomputed within an image and one or more votes at each pixel areaccumulated by adding the summation of the magnitude within a selectedregion. Mean shift clustering may be used to find the local centers inthe region, with the local centers representing actual nuclearlocations. Nuclei detection based on radial symmetry voting is executedon color image intensity data and makes explicit use of the a prioridomain knowledge that the nuclei are elliptical shaped blobs withvarying sizes and eccentricities. To accomplish this, along with colorintensities in the input image, image gradient information is also usedin radial symmetry voting and combined with an adaptive segmentationprocess to precisely detect and localize the cell nuclei. A “gradient”as used herein is, for example, the intensity gradient of pixelscalculated for a particular pixel by taking into consideration anintensity value gradient of a set of pixels surrounding said particularpixel. Each gradient may have a particular “orientation” relative to acoordinate system whose x- and y-axis are defined by two orthogonaledges of the digital image. For instance, nuclei seed detection involvesdefining a seed as a point which is assumed to lie inside a cell nucleusand serve as the starting point for localizing the cell nuclei. Thefirst step is to detect seed points associated with each cell nucleiusing a highly robust approach based on the radial symmetry to detectelliptical-shaped blobs, structures resembling cell nuclei. The radialsymmetry approach operates on the gradient image using a kernel basedvoting procedure. A voting response matrix is created by processing eachpixel that accumulates a vote through a voting kernel. The kernel isbased on the gradient direction computed at that particular pixel and anexpected range of minimum and maximum nucleus size and a voting kernelangle (typically in the range [π/4, π/8]). In the resulting votingspace, local maxima locations that have a vote value higher than apredefined threshold value are saved out as seed points. Extraneousseeds may be discarded later during subsequent segmentation orclassification processes. Other methods are discussed in US PatentPublication No. 2017/0140246, the disclosure of which is incorporated byreference herein for all purposes.

Nuclei may be identified using other techniques known to those ofordinary skill in the art. For example, an image magnitude may becomputed from a particular image channel of one of the H&E or IHCimages, and each pixel around a specified magnitude may be assigned anumber of votes that is based on a summation of the magnitude within aregion around the pixel. Alternatively, a mean shift clusteringoperation may be performed to find the local centers within a votingimage, which represents the actual location of the nucleus. In otherembodiments, nuclear segmentation may be used to segment the entirenucleus based on the now-known centers of the nuclei via morphologicaloperations and local thresholding. In yet other embodiments, model basedsegmentation may be utilized to detect nuclei (i.e. learning the shapemodel of the nuclei from a training data set and using that as the priorknowledge to segment the nuclei in the testing image).

In some embodiments, the nuclei are then subsequently segmented usingthresholds individually computed for each nucleus. For example, Otsu'smethod may be used for segmentation in a region around an identifiednucleus since it is believed that the pixel intensity in the nuclearregions varies. As will be appreciated by those of ordinary skill in theart, Otsu's method is used to determine an optimal threshold byminimizing the intra-class variance and is known to those of skill inthe art. More specifically, Otsu's method is used to automaticallyperform clustering-based image thresholding or, the reduction of a graylevel image to a binary image. The algorithm assumes that the imagecontains two classes of pixels following a bi-modal histogram(foreground pixels and background pixels). It then calculates theoptimum threshold separating the two classes such that their combinedspread (intra-class variance) is minimal, or equivalent (because the sumof pairwise squared distances is constant), so that their inter-classvariance is maximal.

In some embodiments, the systems and methods further compriseautomatically analyzing spectral and/or shape features of the identifiednuclei in an image for identifying nuclei of non-tumor cells. Forexample, blobs may be identified in the first digital image in a firststep. A “blob” as used herein can be, for example, a region of a digitalimage in which some properties, e.g. the intensity or grey value, areconstant or vary within a prescribed range of values. All pixels in ablob can be considered in some sense to be similar to each other. Forexample, blobs may be identified using differential methods which arebased on derivatives of a function of position on the digital image, andmethods based on local extrema. A nuclear blob is a blob whose pixelsand/or whose outline shape indicate that the blob was probably generatedby a nucleus stained with the first stain. For example, the radialsymmetry of a blob could be evaluated to determine if the blob should beidentified as a nuclear blob or as any other structure, e.g. a stainingartifact. For example, in case a blob has a lengthy shape and is notradially symmetric, said blob may not be identified as a nuclear blobbut rather as a staining artifact. Depending on the embodiment, a blobidentified to be a “nuclear blob” may represent a set of pixels whichare identified as candidate nuclei and which may be further analyzed fordetermining if said nuclear blob represents a nucleus. In someembodiments, any kind of nuclear blob is directly used as an “identifiednucleus.” In some embodiments, filtering operations are applied on theidentified nuclei or nuclear blobs for identifying nuclei which do notbelong to biomarker-positive tumor cells and for removing saididentified non-tumor nuclei from the list of already identified nucleior not adding said nuclei to the list of identified nuclei from thebeginning. For example, additional spectral and/or shape features of theidentified nuclear blob may be analyzed to determine if the nucleus ornuclear blob is a nucleus of a tumor cell or not. For example, thenucleus of a lymphocyte is larger than the nucleus of other tissue cell,e.g. of a lung cell. In case the tumor cells are derived from a lungtissue, nuclei of lymphocytes are identified by identifying all nuclearblobs of a minimum size or diameter which is significantly larger thanthe average size or diameter of a normal lung cell nucleus. Theidentified nuclear blobs relating to the nuclei of lymphocytes may beremoved (i.e., “filtered out from”) the set of already identifiednuclei. By filtering out the nuclei of non-tumor cells, the accuracy ofthe method may be increased. Depending on the biomarker, also non-tumorcells may express the biomarker to a certain extent, and may thereforeproduce an intensity signal in the first digital image which does notstem from a tumor cell. By identifying and filtering out nuclei which donot belong to tumor cells from the totality of the already identifiednuclei, the accuracy of identifying biomarker-positive tumor cells maybe increased. These and other methods are described in US PatentPublication 2017/0103521, the disclosure of which is incorporated byreference herein in its entirety for all purposes. In some embodiments,once the seeds are detected, a locally adaptive thresholding method maybe used, and blobs around the detected centers are created. In someembodiments, other methods may also be incorporated, such as markerbased watershed algorithms can also be used to identify the nuclei blobsaround the detected nuclei centers. These and other methods aredescribed in co-pending application PCT/EP2016/051906, published asWO2016/120442, the disclosure of which is incorporated by referenceherein in its entirety for all purposes.

Following detection of the nuclei, features (or metrics) are derivedfrom within the input image. The derivation of metrics from nuclearfeatures are well known in the art and any nuclear features known may beused in the context of the present disclosure. Non-limiting examples ofmetrics that may be computed include:

(A) Metrics Derived from Morphology Features

A “morphology feature” as used herein is, for example, a feature beingindicative of the shape or dimensions of a nucleus. Without wishing tobe bound by any particular theory, it is believed that morphologicalfeatures provide some vital information about the size and shape of acell or its nucleus. For example, a morphology feature may be computedby applying various image analysis algorithms on pixels contained in orsurrounding a nuclear blob or seed. In some embodiments, the morphologyfeatures include area, minor, and major axis lengths, perimeter, radius,solidity, etc. At the cellular level, such features are used to classifya nucleus as belonging to a healthy or diseased cell. At the tissuelevel, the statistics of these features over the tissue are exploited inthe classification of a tissue as diseased or not.

(B) Metrics Derived from Appearance Features

An “appearance feature” as used herein is, for example, a feature havingbeen computed for a particular nucleus by comparing pixel intensityvalues of pixels contained in or surrounding a nuclear blob or seed usedfor identifying the nucleus, whereby the compared pixel intensities arederived from different image channels (e.g. a background channel, achannel for the staining of a biomarker, etc.). In some embodiments, themetrics derived from appearance features are computed from percentilevalues (e.g. the 10th, 50th, and 95th percentile values) of pixelintensities and of gradient magnitudes computed from different imagechannels. For example, at first, a number P of X-percentile values(X=10, 50, 95) of pixel values of each of a plurality IC of imagechannels (e.g. three channels: HTX, DAB, luminance) within a nuclearblob representing the nucleus of interest are identified. Computingappearance feature metrics may be advantageous since the derived metricsmay describe the properties of the nuclear regions as well as describethe membrane region around the nuclei.

(C) Metrics Derived from Background Features

A “background feature” is, for example, a feature being indicative ofthe appearance and/or stain presence in cytoplasm and cell membranefeatures of the cell comprising the nucleus for which the backgroundfeature was extracted from the image. A background feature and acorresponding metrics can be computed for a nucleus and a correspondingcell depicted in a digital image e.g. by identifying a nuclear blob orseed representing the nucleus; analyzing a pixel area (e.g. a ribbon of20 pixels—about 9 microns—thickness around the nuclear blob boundary)directly adjacent to the identified set of cells are computed in,therefore capturing appearance and stain presence in cytoplasm andmembrane of the cell with this nucleus together with areas directlyadjacent to the cell. These metrics are similar to the nuclearappearance features, but are computed in a ribbon of about 20 pixels(about 9 microns) thickness around each nucleus boundary, thereforecapturing the appearance and stain presence in the cytoplasm andmembrane of the cell having the identified nucleus together with areasdirectly adjacent to the cell. Without wishing to be bound by anyparticular theory, the ribbon size is selected because it is believedthat it captures a sufficient amount of background tissue area aroundthe nuclei that can be used to provide useful information for nucleidiscrimination. These features are similar to those disclosed by “J.Kong, et al., “A comprehensive framework for classification of nuclei indigital microscopy imaging: An application to diffuse gliomas,” in ISBI,2011, pp.2128-2131” the disclosure of which is incorporated by referencein its entirety herein for all purposes. It is believed that thesefeatures may be used to determine whether the surrounding tissue isstroma or epithelium (such as in H&E stained tissue samples). Withoutwishing to be bound by any particular theory, it is believed that thesebackground features also capture membrane staining patterns, which areuseful when the tissue samples are stained with appropriate membranestaining agents.

(D) Metrics Derived from Color

In some embodiments, metrics derived from color include color ratios,R/(R+G+B). or color principal components. In other embodiments, metricsderived from color include local statistics of each of the colors(mean/median/variance/std dev) and/or color intensity correlations in alocal image window.

(E) Metrics Derived from Intensity Features

The group of adjacent cells with certain specific property values is setup between the dark and the white shades of grey colored cellsrepresented in a histopathological slide image. The correlation of thecolor feature defines an instance of the size class, thus this way theintensity of these colored cells determines the affected cell from itssurrounding cluster of dark cells.

(F) Metris Derived from Texture Features

Examples of texture features and methods of their derivation aredescribed in PCT Publication Nos. WO/2016/075095 and WO/2016/075096, thedisclosures of which is incorporated by reference herein in theirentireties for all purposes.

(G) Metrics Derived from Spatial Features

In some embodiments, spatial features include a local density of cells;average distance between two adjacent detected cells; and/or distancefrom a cell to a segmented region.

(H) Metrics Derived from Nuclear Features

The skilled artisan will also appreciate that metrics may also bederived from nuclear features. The computation of such nuclear featuresis described by Xing et al. “Robust Nucleus/Cell Detection andSegmentation in Digital Pathology and Microscopy Images: A ComprehensiveReview,” IEEE Rev Biomed Eng 9, 234-263, January 2016, the disclosure ofwhich is hereby incorporated by reference herein in its entirety for allpurposes.

Of course, other features, as known to those of ordinary skill in theart, may be considered and used as the basis for computation offeatures.

By way of another example, cells may be classified as lymphocytes, suchas described in PCT Publication No. WO/2016/075096, the disclosure ofwhich is incorporated by reference herein in its entirety for allpurposes. In particular, PCT Publication No. WO/2016/075096 describes acomputer-implemented method of classifying cells within an image of atissue sample stained in an IHC assay for the presence of a PD-L1biomarker comprising computing nuclear feature metrics from features ofnuclei within the image of the tissue sample; computing contextualinformation metrics based on nuclei of interest with the image of thetissue sample; and classifying the cells within the image of the tissuesample using a combination of the nuclear feature metrics and contextualinformation metrics (as input of the classifier), wherein the cells areclassified as at least one of positive immune cells, positive tumorcells, negative immune cells, and negative tumor cells, or other cells.In some embodiments, the method further comprises the step of creating aforeground segmentation mask to identify individual nuclei within thecells. The publication further describes that, in the context ofPD-L1-stained tissue, regions with lymphocytes that do not express thePD-L1 biomarker (“negative lymphocytes”) are characterized by small blueblobs; regions with lymphocytes that do express the PD-L1 biomarker(“positive lymphocytes”) are characterized by small blue blobs and brownblobs; tumor regions with cells predominantly expressing the PD-L1biomarker (“positive tumor cells”) are characterized by large blue blobsand brown rings; and tumor regions where cells do not express the PD-L1biomarker (“negative tumor cells”) are characterized by large blue blobsonly.

(2) Classification Module

After image feature metrics are derived using the feature extractionmodule 205, the image feature metrics may be used alone or inconjunction with training data (e.g. during training, example cells arepresented together with a ground truth identification provided by anexpert observer according to procedures known to those of ordinary skillin the art) to classify nuclei or cells (using the classification module206). In some embodiments, the system can include a classifier that wastrained based at least in part on a set of training or reference slidesfor each biomarker. The skilled artisan will appreciate that differentsets of slides can be used to train a classifier for each biomarker.Accordingly, for a single biomarker, a single classifier is obtainedafter training. The skilled artisan will also appreciate that sincethere is variability between the image data obtained from differentbiomarkers, a different classifier can be trained for each differentbiomarker so as to ensure better performance on unseen test data, wherethe biomarker type of the test data will be known. The trainedclassifier can be selected based at least in part on how best to handletraining data variability, for example, in tissue type, stainingprotocol, and other features of interest, for slide interpretation.

In some embodiments, the classification module 206 comprises a SupportVector Machine (“SVM”). In general, a SVM is a classification technique,which is based on statistical learning theory where a nonlinear inputdata set is converted into a high dimensional linear feature space viakernels for the non-linear case. Without wishing to be bound by anyparticular theory, it is believed that support vector machines project aset of training data, E, that represents two different classes into ahigh-dimensional space by means of a kernel function, K. In thistransformed data space, nonlinear data are transformed so that a flatline can be generated (a discriminating hyperplane) to separate theclasses so as to maximize the class separation. Testing data are thenprojected into the high-dimensional space via K, and the test data areclassified on the basis of where they fall with respect to thehyperplane. The kernel function K defines the method in which data areprojected into the high-dimensional space.

In other embodiments, classification is performed using an AdaBoostalgorithm. The AdaBoost is an adaptive algorithm which combines a numberof weak classifiers to generate a strong classifier. Image pixelsidentified by a pathologist during the training stage (e.g. those havinga particular stain or belonging to a particular tissue type) are used togenerate probability density functions for each of the individualtexture features Φj, for j∈{1, . . . , K} which are considered as weakclassifiers. Bayes Theorem is then used to generate likelihood scenesLj=(Cj, 1 j∈{1, . . . , K}) for each Φj which constitute the weaklearners. These are combined by the AdaBoost algorithm into a strongclassifier Πj=ΣTi=1αjilji where for every pixel cj∈Cj, Πj (cj) is thecombined likelihood that pixel cj belongs to class ωT, where αji is theweight determined during training for feature Φi, and T is the number ofiterations.

(3) Scoring Module

Following the derivation of image feature metrics and/or classificationof the cells/nuclei, a scoring module 207 is utilized to provide anexpression score based on the derived image feature metrics.Non-limiting examples of diagnostic metrics include, without limitation,percent positivity, an Allred score, an immunohistochemistry combinationscore, or an H-score. The skilled artisan will appreciate that any ofthese examples of expression scores may be used as the deriveddiagnostic feature (step 320). In some embodiments, expression scoresfrom the scoring module are utilized as diagnostic metrics.

For example, the feature extraction module 205 may comprise a series ofimage analysis algorithms and be used to determine a presence of one ormore of a nucleus, a cell wall, a tumor cell, or other structures withinthe identified cell clusters, as described herein. In some embodiments,derived stain intensity values and counts of specific nuclei for eachfield of view may be used to determine various marker expression scores,such as percent positivity or an H-Score. Methods for scoring aredescribed in further detail in commonly-assigned and co-pendingapplications WO/2014/102130A1 “Image analysis for breast cancerprognosis” filed Dec. 19, 2013, and WO/2014/140085A1 “Tissueobject-based machine learning system for automated scoring of digitalwhole slides”, filed Mar. 12, 2104, the contents of each are herebyincorporated by reference in their entirety herein for all purposes.

By way of example, automated image analysis algorithms in the scoringmodule 207 may be used to interpret each one of the IHC slides in theseries to detect tumor nuclei that are positively and negatively stainedfor a particular biomarker, such as Ki67, ER, PR, HER2, etc. Based onthe detected positive and negative tumor nuclei, various slide levelscores such as marker percent positivity, H-scores, etc. may be computedusing one or more methods.

In some embodiments, the expression score is an H-score. In someembodiments, the ‘H’ score is used to assess the percentage of tumorcells with cell membrane staining graded as ‘weak,’ ‘moderate’ or‘strong.’ The grades are summated to give an overall maximum score of300 and a cut-off point of 100 to distinguish between a ‘positive’ and‘negative.’ For example, a membrane staining intensity (0, 1+, 2+, or3+) is determined for each cell in a fixed field of view (or here, eachcell in a tumor or cell cluster). The H-score may simply be based on apredominant staining intensity, or more complexly, can include the sumof individual H-scores for each intensity level seen. By one method, thepercentage of cells at each staining intensity level is calculated, andfinally, an H-score is assigned using the following formula: [1×(% cells1+)+2×(% cells 2+)+3×(% cells 3+)]. The final score, ranging from 0 to300, gives more relative weight to higher-intensity membrane staining ina given tumor sample. The sample can then be considered positive ornegative on the basis of a specific discriminatory threshold. Additionalmethods of calculating an H-score are described in United States PatentPublication 2015/0347702, the disclosure of which is hereby incorporatedby reference herein in its entirety for all purposes.

In some embodiments, the expression score is an Allred score. The Allredscore is a scoring system which looks at the percentage of cells thattest positive for hormone receptors, along with how well the receptorsshow up after staining (this is called “intensity”). This information isthen combined to score the sample on a scale from 0 to 8. The higher thescore, the more receptors are found and the easier they are to see inthe sample.

In other embodiments, the expression score is percent positivity. Again,in the context of scoring a breast cancer sample stained for the PR andKi-67 biomarkers, for the PR and Ki-67 slides, the percent positivity iscalculated (e.g., the total number of nuclei of cells (e.g., malignantcells) that are stained positive in each field of view in the digitalimage of a slide are summed and divided by the total number ofpositively and negatively stained nuclei from each of the fields of viewof a digital image) in a single slide as follows: Percentpositivity=number of positively stained cells/(number of positivelystained cells+number of negatively stained cells).

In other embodiments, the expression score is an immunohistochemistrycombination score, which is a prognostic score based on a number of IHCmarkers, wherein the number of markers is greater than one. IHC4 is onesuch score based on four measured IHC markers, namely ER, HER2, Ki-67,and PR in a breast cancer sample (for example see Cuzick et al., J.Clin. Oncol. 29:4273-8, 2011, and Barton et al., Br. J. Cancer 1-6, Apr.24, 2012, both herein incorporated by reference for all purposes). Inone example, and in the context of detecting expression scores forbreast cancer, an IHC4 score is calculated using, for example, thefollowing formula:

IHC4=94.7×{−0.100ER10−0.079PR10+0.586HER2+0.240 ln(1+10×Ki67)}.

(4) Multivariate Cox Model Module

In some embodiments, if multiple diagnostic features are selected (suchas by a pathologist), a multivariate Cox model may be constructed toyield a diagnostic feature metric (step 320) that comprises a weighedcombination of multiple metrics (e.g. multiple image feature metricsand/or expression scores). In some embodiments, the multiple diagnosticfeature metrics are derived using the feature extraction module 205, theclassification module 206, and/or scoring module 207. For example, themultiple metrics may be image feature metrics or expression scores.Again, by way of example, the Cox model module may weight and combine anH-score and a biomarker percent positivity. Alternatively, multiplemetrics may be combined where the multiple metrics are determinedthrough a machine learning method using the prognostic featurederivation module 209.

Cox's proportional hazards model is analogous to a multiple regressionmodel and enables the difference between, for example, survival times ofparticular groups of patients to be tested while allowing for otherfactors. In general, it is a survival analysis regression model, whichdescribes the relation between the event incidence, as expressed by thehazard function and a set of covariates. In this model, the response(dependent) variable is the ‘hazard’. The hazard is the instantaneousevent probability at a given time, or the probability that an individualunder observation experiences the event in a period centered around thatpoint in time. In the context of survival analysis, the hazard is theprobability of dying given that patients have survived up to a givenpoint in time, or the risk for death at that moment.

Mathematically, the Cox model is written as:

h(x,t)=h ₀(t)×exp{b ₁ x ₁ +b ₂ x ₂ + . . . +b _(p) x _(p)}

where the hazard function h(x,t) is dependent on (or determined by) aset of p covariates (x₁, x₂, . . . , x_(p)), whose impact is measured bythe size of the respective coefficients (b₁, b₂, . . . , b_(p)). Theterm ho is called the baseline hazard, and is the value of the hazard ifall the x_(i) are equal to zero (the quantity exp(0) equals 1). The ‘t’in h(t) reminds us that the hazard may (and probably will) vary overtime. An appealing feature of the Cox model is that the baseline hazardfunction is estimated non-parametrically, and so unlike most otherstatistical models, the survival times are not assumed to follow aparticular statistical distribution. As applied here, the p covariatesare the various diagnostic features values under consideration.

The Cox model is essentially a multiple linear regression of thelogarithm of the hazard on the variables x_(i), with the baseline hazardbeing an ‘intercept’ term that varies with time. The covariates then actmultiplicatively on the hazard at any point in time, and this providesus with the key assumption of the PH model: the hazard of the event inany group is a constant multiple of the hazard in any other. Thisassumption implies that the hazard curves for the groups should beproportional and cannot cross. Proportionality implies that thequantities exp(b_(i)) are called hazard ratios. A value of b_(i) greaterthan zero, or equivalently a hazard ratio greater than one, indicatesthat as the value of the i-th covariate increases, the event hazardincreases and thus the length of survival decreases. Put another way, ahazard ratio above 1 indicates a covariate that is positively associatedwith the event probability, and thus negatively associated with thelength of survival.

In some embodiments, for each patient sample of the test cohort, data isobtained regarding the outcome being tracked (time to death, time torecurrence, or time to progression) and the feature metric for eachbiomarker being analyzed. Candidate Cox proportional models aregenerated by entering the diagnostic feature data and survival data foreach individual of the cohort into a computerized statistical analysissoftware suite (such as The R Project for Statistical Computing(available at https://www.r-project.org/), SAS, MATLAB, among others).

Following construction of the Cox model, the hazard function (h(x,t))may be used as a diagnostic feature (step 320).

(5) Prognostic Feature Derivation Module

In some embodiments, a machine learning algorithm may be utilized todetermine a set of image feature metrics that are most relevant inpredicting a patient outcome. Said another way, rather than usepre-specified image expression scores or image feature metrics (or anycombination thereof, such as combined in a multivariate Cox model), amachine learning technique may be used to discover features which may bemore accurate in predicting a response, i.e. a data-driven diagnosticfeature discovery method. In some embodiments, prognostic featurederivation module 209 may be utilized to validate the predictive valueof novel image feature metrics.

In some embodiments, the input data (image data and patient data) may bederived from a placebo arm of a clinical trial (either Phase II and/orPhase III), and a classifier may be used to determine those image andclinical features which are believed to be important in a binarycategorization of patients. In other embodiments, the input data (imagedata and patient data) may be derived from a drug study arm of aclinical trial (either Phase II and/or Phase III), and a classifier maybe used to determine those image and clinical features which arebelieved to be important in a binary categorization of patients. Forexample, input images from a patient population may comprise 100 imagefeature metrics after image analysis. A machine learning algorithm maybe utilized to determine which of those 100 image feature metrics bestpredict the patient population outcome and, by way of example, this maybe 10 top image feature metrics from the 100 total image featuremetrics. Again, by way of example, these 10 top image feature metricsmay be combined using a multivariate Cox hazard model to provide adiagnostic feature that may be utilized for further downstreamprocessing.

In some embodiments, a diagnostic feature metric (step 320) isdetermined using a machine-based learning technique. In someembodiments, a binary classification problem is created to train theclassifier. With reference to FIG. 5, patients are split into low andhigh survival groups (step 500). The determination of which patients ina cohort fall into a particular group may be based, for example, on athreshold overall survival time, such as a threshold of a predeterminednumber of months, e.g. 7-months. More specifically, a particular cohortof patients may be classified into (i) those that survived less than amedian survival time (e.g. those that survived less than a predeterminedmedian OS); and (ii) those that survived more than a median survivaltime (e.g. those that survived more than a predetermined median OS).Then a classifier is built (step 510) using the image feature metricsderived at step 310.

Once trained, the classifier may then be utilized to determine thoseimage features in test images that best correspond to certain patientoutcome data. As noted above, such prognostic features differ frompre-specified diagnostic features (such as H-score) in that that imageand clinical features are derived from image data that most accuratelystratifies patient populations or cohorts. The prognostic feature set isthen supplied to the multivariate Cox model module 208 to provide adiagnostic feature metric (step 320) for use in downstream processing,i.e. a multivariate Cox model is built using the multiple prognosticfeatures from the classifier.

Cutoff Determination Module

In medical research and, in particular, cancer research, when a survivalanalysis is conducted, it is a common practice to dichotomize acontinuous covariate. In some embodiments, a derived diagnostic feature(from step 320) and patient outcome data (from database 212) may be usedas input in a bio-statistical analysis to find an optimal cutoff for ascoring diagnostic. In view of this, the diagnostic feature from step320 is then provided, along with patient outcome data (from database212) to the cutoff determination module 210 such that a cutoff point orcut point (used interchangeably herein) may be determined (step 330).

In some embodiments, a log rank statistic minimization method isutilized to determine a cutoff value. Compared to data dependentmethods, a long rank statistic based method finds a statisticallyoptimal solution. The log rank statistic method is a commonly usedprocedure for comparing two survival distributions (e.g. a placebocohort and a treatment cohort). It is a nonparametric test and it isbelieved that the method is appropriate to use when the data are rightskewed and censored.

Let R be the risk factor of interest measured as a continuous variableand T be the outcome variable. In case of survival analysis, the outcomeof interest T, is oftentimes time to death but it can also be time tosome other event of interest. In some embodiments, the population isdivided into two groups based on the cut point: subjects with the valueof the risk factor less than or equal to the value of the cut point andsubjects with the value of the risk factor greater than the cut point.Let t(1)<t(2)< . . . <t(k) be the ordered observed event times of theoutcome variable T. Let C be the set of K distinct values of thecontinuous covariate R. Then, based on one hypothetical cut point fromC, let d(i) be the number of events at time t(1), r(i) be the number ofsubjects at risk prior to time t(i) and d(i)+ and r(i)+ be the number ofevents at time t(i) in group R>C and number of subjects at risk justprior to t(i) in the group R>C. Similarly, d(i)− and r(i)− be the numberof events at time t(i) in group R less than or equal to C and number ofsubjects at risk just prior to t(i) in the group R than or equal to C.Thus, the log rank statistic for some fixed C is given by:

${{Log}\mspace{14mu} {Rank}\mspace{14mu} {Statistic}} = {{L_{k}(t)} = {\sum\limits_{i = 1}^{k}( {d_{i}^{+} - {d_{i}\frac{r_{i}^{+}}{r_{i}}}} )}}$

The optimal cut point is that value of C, Ck that maximizes the absolutevalue of Lk (t). Ck therefore gives the value of the continuouscovariate that gives the maximum difference between the subjects in thetwo groups defined by the cut point. The log rank statistic and othermethods of determining cut points are described by L. Zhang and Y. Shu,“Advances in Visual Computing,” 12th International Symposium, ISVC 2016,Dec. 12-14, 2016, Proceedings, Part 1, pp.57-63, the disclosure of whichis hereby incorporated by reference herein in its entirety for allpurposes. The optimal cut point will be the threshold on the diagnosticto identify a patient as positive or negative for the test.

Once the optimal cutoff point is determined (step 330), patients may bestratified into diagnostic positive and diagnostic negative groups (step340). This automatic diagnostic cut point and the resultantstratification can, in some embodiments, be compared against a manuallyselected diagnostic cut point and stratification (step 350). In someembodiments, the comparison may assist in determining whether thecorrect companion diagnostic was used, or if the threshold set in aclinical trial was too high or too low, i.e. the impact of the manuallyselected diagnostic cut point may be determined (step 360).

(1) Drug Response Curve Generation Module

In some embodiments, drug response curves are generated (step 370) basedon the stratified patient groups (from step 350). In some embodiments,the generated drug response curves may be used to determine if clinicalefficacy exists in a Phase II or Phase III trial, the drug responsecurves used to compare control and drug study cohorts in either trial.In some embodiments, drug response curves are generated for both theplacebo and the drug study cohorts.

In analyzing survival data, two functions that are dependent on time areof particular interest: the survival function and the hazard function.The survival function S(t) is defined as the probability of surviving atleast to time t. The hazard function h(t) is the conditional probabilityof dying at time t having survived to that time.

The graph of S(t) against t is called the survival curve. TheKaplan-Meier method can be used to estimate this curve from the observedsurvival times without the assumption of an underlying probabilitydistribution. The method is based on the basic idea that the probabilityof surviving k or more periods from entering the study is a product ofthe k observed survival rates for each period (i.e. the cumulativeproportion surviving), given by the following:

S(k)=p ₁ ×p ₂ ×p ₃ × . . . ×p _(k)

Here, p₁ is the proportion surviving the first period, p₂ is theproportion surviving beyond the second period conditional on havingsurvived up to the second period, and so on. The proportion survivingperiod i having survived up to period i is given by:p_(i)=(r_(i)−d_(i))/r_(i) where r_(i) is the number alive at thebeginning of the period and d_(i) the number of deaths within theperiod.

As such, a Kaplan-Meier curve is a statistical picture of the percentageof patients surviving over a period of time; it cannot be summed up witha single number such as median survival or a landmark measure (i.e. ameasure of the number of people alive at a predetermined time). Theslope of the curve is the overall rate of death or risk of death; thisis called the hazard ratio or the hazard rate. For example, if a studyhas two arms (e.g. a placebo group and a group administered a drug),then two survival curves can be constructed, each with its own hazardrate. The hazard ratio (“HR”) is the ratio of the hazard for the studydrug group divided by the hazard for the placebo control group. If HR=1,the hazard or risk of death in the two groups is equal. If HR>1, therisk of death is increased in the study group compared with the controlgroup, while HR<1 means the risk of death is decreased in the studygroup compared with the control group.

In some embodiments, the larger the separation of the curves the greaterdifference between the treatment groups in the endpoint being analyzed.If the treatment arms represented in the Kaplan-Meier curve follow asimilar path, it suggests that there is only a small amount ofdifference between the arms of the study in the endpoint being measured.If the arms were to meet, that would mean at that particular time pointthere was no difference between the two arms of the study in theendpoint being measured.

FIG. 6 provides an illustrative example of a Kaplan-Meier curve to showthe difference between the median and landmark overall survivalmeasurements of two regimens, namely Regimen A and Regimen B (whichcould be a placebo group vs. a drug study group). In this particularexample, the difference in median survival times appears to berelatively small, as does the difference in overall survival at one-year(93% vs 85%), whereas at five years the overall survival difference is23% vs 12%. Because of the shape of the curves, however, there also maybe a large difference in overall survival as measured by the hazardratio.

Cohort Signature Module

As a byproduct of the whole tumor image analysis, a large quantity ofimage feature metrics are computed for each patient tissue slide. Ifgenomic analysis of the patient's tissue sample is also performed,similarly a set of molecular and genomic variants are output for eachpatient (tissue analysis data). A feature vector may be generated bycombining the image feature metrics and the genomic features/tissueanalysis from each patient along with certain clinical attributes. Insome embodiments, from the generated feature vectors of all of thepatients in a given cohort, a feature matrix may be constructed. In someembodiments, a statistical analysis of the feature matrix (e.g.principal component analysis, hierarchical clustering of featuresfollowed up with feature selection) will yield a condensed featurematrix for the cohort called a “cohort signature,” i.e. a matrix thatcharacterizes the tissue feature variation along with featurecorrelations. In some embodiments, the statistical analysis allows oneto determine whether two given cohorts are similar or different.Specifically, the statistical analysis can facilitate a determine of howsimilar or how different two datasets are, e.g. how similar or howdifferent Phase II and Phase III cohorts are.

With reference to FIG. 7, data is received or acquired from each of twodifferent patient populations. In some embodiments, image datacorresponding to biological samples from two different patientpopulations is received as input (steps 700 and 701). In someembodiments, the biological samples are stained with a primary stain anda counterstain (i.e. an H&E image) or stained for the presence of aparticular biomarker (i.e. and IHC image). In addition to the imagedata, other tissue analysis data is also received for each patient inthe patient populations. In some embodiments, the tissue analysis datais molecular or genomic data (such as genomic data stored in database212).

Next, image feature metrics are computed (steps 710 and 711) for each ofthe images received as input and for each patient population. In someembodiments, the image feature metrics are computed using featureextraction module 205, the classification module 206, and/or the scoringmodule 207 and as described herein.

The computed image feature metrics are then combined with the tissueanalysis data (steps 720 and 721) for each patient to provide a set oftissue features for each patient (e.g. a feature vector) in the patientpopulations. The tissue feature data together with clinical attributesfor all patients, may then be utilized to generate a correlation matrixfor each patient population. (steps 740 and 741). In some embodiments,the clinical attributes are age, weight, sex, ethnicity, etc. Thecorrelation matrices for each of the two patient populations may then becompared using statistical methods to determine how similar or howdifferent the patient populations are from each other (step 750).

In addition, distributions of individual tissue features may be computedfor each of the patient populations (steps 730 and 731). The computeddistributions may then be compared using statistical methods to againdetermine how similar or different the patient populations are to eachother.

In some embodiments, the first patient population is a Phase II patientcohort, while the second patient population is a Phase III patientcohort. In some embodiments, the first patient population is a Phase IIplacebo cohort, while the second patient population is a Phase IIIplacebo cohort. In some embodiments, the first patient population is aPhase II test arm cohort, while the second patient population is a PhaseIII placebo cohort. In some embodiments, the first patient population isa Phase II test arm cohort, while the second patient population is aPhase III test arm cohort. In some embodiments, the first patientpopulation is a Phase II placebo cohort, while the second patientpopulation is a Phase II test arm cohort. In some embodiments, the firstpatient population is data collected pre-treatment; while the secondpatient population is data collected from the same patientspost-treatment.

Other Components for Practicing Embodiments of the Present Disclosure

Other components (e.g. systems or modules) are described below which maybe used in conjugation with the systems and methods of the presentdisclosure.

(1) Unmixing Module

In some embodiments, the images received as input may be multipleximages, i.e. the image received is of a biological sample stained withmore than one stain. In these embodiments, and prior to furtherprocessing, the multiple image is first unmixed into its constituentchannels, where each unmixed channel corresponds to a particular stainor signal. In some embodiments, the unmixed images (often referred to as“channel images” or “image channel images”) and may be used as the inputfor each module described herein. For example, inter-markerheterogeneity may be determined with a first H&E image, a secondmultiplex image stained for a plurality of cluster of differentiationmarkers (CD3, CD8, etc.), and a plurality of simplex images each stainedfor a particular biomarker (e.g. ER, PR, Ki67, etc.). In this example,the multiplex image is first unmixed into its constituent channelimages, and those channel images may be used along with the H&E imageand the plurality of simplex images to determined inter-markerheterogeneity.

In some embodiments, in a sample comprising one or more stains andhematoxylin, individual images may be produced for each channel of theone or more stains and hematoxylin. Without wishing to be bound by anyparticular theory, it is believed that these channels highlightdifferent tissue structures in the tissue image, thus, they may bereferred to as structural image channels. In some embodiments, unmixingprovides at least a hematoxylin image channel image. In someembodiments, an acquired image is unmixed into a separate channelrepresenting the local amounts of hematoxylin and highlighting nucleiregions within the image. The skilled artisan will appreciate thatfeatures extracted from these channels are useful in describing thedifferent biological structures present within any image of a tissue.

The multi-spectral image provided by the imaging acquisition module 202is a weighted mixture of the underlying spectral signals associated theindividual biomarkers and noise components. At any particular pixel, themixing weights are proportional to the biomarker expressions of theunderlying co-localized biomarkers at the particular location in thetissue and the background noise at that location. Thus, the mixingweights vary from pixel to pixel. The spectral unmixing methodsdisclosed herein decompose the multi-channel pixel value vector at eachand every pixel into a collection of constituent biomarker end membersor components and estimate the proportions of the individual constituentstains for each of the biomarkers.

Unmixing is the procedure by which the measured spectrum of a mixedpixel is decomposed into a collection of constituent spectra, orendmembers, and a set of corresponding fractions, or abundances, thatindicate the proportion of each endmember present in the pixel.Specifically, the unmixing process can extract stain-specific channelsto determine local concentrations of individual stains using referencespectra that are well known for standard types of tissue and staincombinations. The unmixing may use reference spectra retrieved from acontrol image or estimated from the image under observation. Unmixingthe component signals of each input pixel enables retrieval and analysisof stain-specific channels, such as a hematoxylin channel and an eosinchannel in H&E images, or a diaminobenzidine (DAB) channel and acounterstain (e.g., hematoxylin) channel in IHC images. The terms“unmixing” and “color deconvolution” (or “deconvolution”) or the like(e.g. “deconvolving,” “unmixed”) are used interchangeably in the art.

In some embodiments, the multiplex images are unmixed with the featureextraction module 205 using liner unmixing. Linear unmixing isdescribed, for example, in ‘Zimmermann “Spectral Imaging and LinearUnmixing in Light Microscopy” Adv Biochem Engin/Biotechnol (2005)95:245-265’ and in in C. L. Lawson and R. J. Hanson, “Solving leastsquares Problems”, Prentice Hall, 1974, Chapter 23, p. 161,’ thedisclosures of which are incorporated herein by reference in theirentirety for all purposes. In linear stain unmixing, the measuredspectrum (S(λ)) at any pixel is considered a linear mixture of stainspectral components and equals the sum of the proportions or weights (A)of each individual stain's color reference (R(λ)) that is beingexpressed at the pixel

S(λ)=A ₁ ×R ₁(λ)+A ₂ ×R ₂(λ)+A ₃ ×R ₃(λ) . . . A _(i) ×R ₁(λ)

which can be more generally expressed as in matrix form as

S(λ)=ΣA _(i) ×R _(i)(λ) or S=R×A

If there are M channels images acquired and N individual stains, thecolumns of the M×N matrix R are the optimal color system as derivedherein, the N×1 vector A is the unknown of the proportions of individualstains and the M×1 vector S is the measured multichannel spectral vectorat a pixel. In these equations, the signal in each pixel (S) is measuredduring acquisition of the multiplex image and the reference spectra,i.e. the optimal color system, is derived as described herein. Thecontributions of various stains (A_(i)) can be determined by calculatingtheir contribution to each point in the measured spectrum. In someembodiments, the solution is obtained using an inverse least squaresfitting approach that minimizes the square difference between themeasured and calculated spectra by solving the following set ofequations,

[∂Σ_(j) {S(λ_(j))−Σ_(i) A _(i) ×R _(i)(λ_(j))}2]/∂A _(i)=0

In this equation, j represents the number of detection channels and iequals the number of stains. The linear equation solution often involvesallowing a constrained unmixing to force the weights (A) to sum tounity.

In other embodiments, unmixing is accomplished using the methodsdescribed in WO2014/195193, entitled “Image Adaptive PhysiologicallyPlausible Color Separation,” filed on May 28, 2014, the disclosure ofwhich is hereby incorporated by reference in its entirety herein for allpurposes. In general, WO2014/195193 describes a method of unmixing byseparating component signals of the input image using iterativelyoptimized reference vectors. In some embodiments, image data from anassay is correlated with expected or ideal results specific to thecharacteristics of the assay to determine a quality metric. In the caseof low quality images or poor correlations against ideal results, one ormore reference column vectors in matrix R are adjusted, and the unmixingis repeated iteratively using adjusted reference vectors, until thecorrelation shows a good quality image that matches physiological andanatomical requirements. The anatomical, physiological, and assayinformation may be used to define rules that are applied to the measuredimage data to determine the quality metric. This information includeshow the tissue was stained, what structures within the tissue wereintended or not intended to be stained, and relationships betweenstructures, stains, and markers specific to the assay being processed.An iterative process results in stain-specific vectors that can generateimages that accurately identify structures of interest and biologicallyrelevant information, are free from any noisy or unwanted spectra, andtherefore fit for analysis. The reference vectors are adjusted to withina search space. The search space defines a range of values that areference vector can take to represent a stain. The search space may bedetermined by scanning a variety of representative training assaysincluding known or commonly occurring problems, and determininghigh-quality sets of reference vectors for the training assays.

In other embodiments, unmixing is accomplished using the methodsdescribed in WO2015/124772, entitled “Group Sparsity Model for ImageUnmixing,” filed on Feb. 23, 2015, the disclosure of which is herebyincorporated by reference in its entirety herein for all purposes. Ingeneral, WO2015/124772 describes unmixing using a group sparsityframework, in which fractions of stain contributions from a plurality ofcolocation markers are modeled within a “same group” and fractions ofstain contributions from a plurality of non-colocation markers aremodeled in different groups, providing co-localization information ofthe plurality of colocation markers to the modeled group sparsityframework, solving the modeled framework using a group lasso to yield aleast squares solution within each group, wherein the least squaressolution corresponds to the unmixing of the colocation markers, andyielding a sparse solution among the groups that corresponds to theunmixing of the non-colocation markers. Moreover, WO2015124772 describesa method of unmixing by inputting image data obtained from thebiological tissue sample, reading reference data from an electronicmemory, the reference data being descriptive of the stain color of eachone of the multiple stains, reading colocation data from electronicmemory, the colocation data being descriptive of groups of the stains,each group comprising stains that can be collocated in the biologicaltissue sample, and each group forming a group for the group lassocriterion, at least one of the groups having a size of two or above, andcalculating a solution of the group lasso criterion for obtaining theunmixed image using the reference data as a reference matrix. In someembodiments, the method for unmixing an image may comprise generating agroup sparsity model wherein a fraction of a stain contribution fromcolocalized markers is assigned within a single group and a fraction ofa stain contribution from non-colocalized markers is assigned withinseparate groups, and solving the group sparsity model using an unmixingalgorithm to yield a least squares solution within each group.

Additional Embodiments

In some embodiments, to retrospectively evaluate the impact of thetreatment along with other patient attributes (tissue, clinical andpathological) a data-driven machine learning approach is disclosed. Thetissue attributes can be a combination of image, genomic and molecularfeatures measured and/or derived from the patient tissue. The machinelearning approach can be a regression model or classifier model. Tobuild the regression model, the complete survival data is used. In theregression model, the model predicts the probability of favorableresponse from a given patient data. To build a classifier model, basedon a user specified threshold value on the patient outcome data (overallsurvival, PFS, RFS, complete response), the patient pool is categorizedinto two patients pools of favorable and unfavorable response groups. Inclassifier model, the classifier predicts whether a particular patientfalls in the favorable or unfavorable response group. And with all thepatient attributes as feature data, a regression or binary classifiermodel. The learnt model, in addition to learning to predict the drugresponse, can discover and output a set of important features,predictive attributes, that explain the observed patient responses inthe study. Thus, it can inform whether if the drug or the treatment is apredictive attribute; and/or if any other patients attributes explainthe patient response and a combination of multiple and interactingfeatures better inform about the cohort result data. And this newinformation, can guide in recognizing a subset of the Phase IIIpopulation for whom the treatment or drug results in a favorableresponse; or else design a different prospective study to account forthe newly discovered set of features.

Other System Components

The digital pathology system 200 of the present disclosure may be tiedto a specimen processing apparatus that can perform one or morepreparation processes on the tissue specimen. The preparation processcan include, without limitation, deparaffinizing a specimen,conditioning a specimen (e.g., cell conditioning), staining a specimen,performing antigen retrieval, performing immunohistochemistry staining(including labeling) or other reactions, and/or performing in situhybridization (e.g., SISH, FISH, etc.) staining (including labeling) orother reactions, as well as other processes for preparing specimens formicroscopy, microanalyses, mass spectrometric methods, or otheranalytical methods.

The processing apparatus can apply fixatives to the specimen. Fixativescan include cross-linking agents (such as aldehydes, e.g., formaldehyde,paraformaldehyde, and glutaraldehyde, as well as non-aldehydecross-linking agents), oxidizing agents (e.g., metallic ions andcomplexes, such as osmium tetroxide and chromic acid),protein-denaturing agents (e.g., acetic acid, methanol, and ethanol),fixatives of unknown mechanism (e.g., mercuric chloride, acetone, andpicric acid), combination reagents (e.g., Carnoy's fixative, methacarn,Bouin's fluid, B5 fixative, Rossman's fluid, and Gendre's fluid),microwaves, and miscellaneous fixatives (e.g., excluded volume fixationand vapor fixation).

If the specimen is a sample embedded in paraffin, the sample can bedeparaffinized using appropriate deparaffinizing fluid(s). After theparaffin is removed, any number of substances can be successivelyapplied to the specimen. The substances can be for pretreatment (e.g.,to reverse protein-crosslinking, expose nucleic acids, etc.),denaturation, hybridization, washing (e.g., stringency wash), detection(e.g., link a visual or marker molecule to a probe), amplifying (e.g.,amplifying proteins, genes, etc.), counterstaining, coverslipping, orthe like.

The specimen processing apparatus can apply a wide range of substancesto the specimen. The substances include, without limitation, stains,probes, reagents, rinses, and/or conditioners. The substances can befluids (e.g., gases, liquids, or gas/liquid mixtures), or the like. Thefluids can be solvents (e.g., polar solvents, non-polar solvents, etc.),solutions (e.g., aqueous solutions or other types of solutions), or thelike. Reagents can include, without limitation, stains, wetting agents,antibodies (e.g., monoclonal antibodies, polyclonal antibodies, etc.),antigen recovering fluids (e.g., aqueous- or non-aqueous-based antigenretrieval solutions, antigen recovering buffers, etc.), or the like.Probes can be an isolated nucleic acid or an isolated syntheticoligonucleotide, attached to a detectable label or reporter molecule.Labels can include radioactive isotopes, enzyme substrates, co-factors,ligands, chemiluminescent or fluorescent agents, haptens, and enzymes.

After the specimens are processed, a user can transport specimen-bearingslides to the imaging apparatus. In some embodiments, the imagingapparatus is a brightfield imager slide scanner. One brightfield imageris the iScan Coreo brightfield scanner sold by Ventana Medical Systems,Inc. In automated embodiments, the imaging apparatus is a digitalpathology device as disclosed in International Patent Application No.:PCT/US2010/002772 (Patent Publication No.: WO/2011/049608) entitledIMAGING SYSTEM AND TECHNIQUES or disclosed in U.S. Patent ApplicationNo. 61/533,114, filed on Sep. 9, 2011, entitled IMAGING SYSTEMS,CASSETTES, AND METHODS OF USING THE SAME. International PatentApplication No. PCT/US2010/002772 and U.S. Patent Application No.61/533,114 are incorporated by reference in their entirety for allpurposes.

The imaging system or apparatus may be a multispectral imaging (MSI)system or a fluorescent microscopy system. The imaging system used hereis an MSI. MSI, generally, equips the analysis of pathology specimenswith computerized microscope-based imaging systems by providing accessto spectral distribution of an image at a pixel level. While thereexists a variety of multispectral imaging systems, an operational aspectthat is common to all of these systems is a capability to form amultispectral image. A multispectral image is one that captures imagedata at specific wavelengths or at specific spectral bandwidths acrossthe electromagnetic spectrum. These wavelengths may be singled out byoptical filters or by the use of other instruments capable of selectinga pre-determined spectral component including electromagnetic radiationat wavelengths beyond the range of visible light range, such as, forexample, infrared (IR).

An MSI system may include an optical imaging system, a portion of whichcontains a spectrally-selective system that is tunable to define apre-determined number N of discrete optical bands. The optical systemmay be adapted to image a tissue sample, illuminated in transmissionwith a broadband light source onto an optical detector. The opticalimaging system, which in one embodiment may include a magnifying systemsuch as, for example, a microscope, has a single optical axis generallyspatially aligned with a single optical output of the optical system.The system forms a sequence of images of the tissue as the spectrallyselective system is being adjusted or tuned (for example with a computerprocessor) such as to assure that images are acquired in differentdiscrete spectral bands. The apparatus may additionally contain adisplay in which appears at least one visually perceivable image of thetissue from the sequence of acquired images. The spectrally-selectivesystem may include an optically-dispersive element such as a diffractivegrating, a collection of optical filters such as thin-film interferencefilters or any other system adapted to select, in response to either auser input or a command of the pre-programmed processor, a particularpass-band from the spectrum of light transmitted from the light sourcethrough the sample towards the detector.

An alternative implementation, a spectrally selective system definesseveral optical outputs corresponding to N discrete spectral bands. Thistype of system intakes the transmitted light output from the opticalsystem and spatially redirects at least a portion of this light outputalong N spatially different optical paths in such a way as to image thesample in an identified spectral band onto a detector system along anoptical path corresponding to this identified spectral band.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Any of the modulesdescribed herein may include logic that is executed by the processor(s).“Logic,” as used herein, refers to any information having the form ofinstruction signals and/or data that may be applied to affect theoperation of a processor. Software is an example of logic.

A computer storage medium can be, or can be included in, acomputer-readable storage device, a computer-readable storage substrate,a random or serial access memory array or device, or a combination ofone or more of them. Moreover, while a computer storage medium is not apropagated signal, a computer storage medium can be a source ordestination of computer program instructions encoded in an artificiallygenerated propagated signal. The computer storage medium can also be, orcan be included in, one or more separate physical components or media(e.g., multiple CDs, disks, or other storage devices). The operationsdescribed in this specification can be implemented as operationsperformed by a data processing apparatus on data stored on one or morecomputer-readable storage devices or received from other sources.

The term “programmed processor” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable microprocessor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus also can include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,subprograms, or portions of code). A computer program can be deployed tobe executed on one computer or on multiple computers that are located atone site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random-access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., an LCD (liquid crystal display), LED(light emitting diode) display, or OLED (organic light emitting diode)display, for displaying information to the user and a keyboard and apointing device, e.g., a mouse or a trackball, by which the user canprovide input to the computer. In some implementations, a touch screencan be used to display information and receive input from a user. Otherkinds of devices can be used to provide for interaction with a user aswell; for example, feedback provided to the user can be in any form ofsensory feedback, e.g., visual feedback, auditory feedback, or tactilefeedback; and input from the user can be received in any form, includingacoustic, speech, or tactile input. In addition, a computer can interactwith a user by sending documents to and receiving documents from adevice that is used by the user; for example, by sending web pages to aweb browser on a user's client device in response to requests receivedfrom the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks). For example,the network 20 of FIG. 1 can include one or more local area networks.

The computing system can include any number of clients and servers. Aclient and server are generally remote from each other and typicallyinteract through a communication network. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

All of the U.S. patents, U.S. patent application publications, U.S.patent applications, foreign patents, foreign patent applications andnon-patent publications referred to in this specification and/or listedin the Application Data Sheet are incorporated herein by reference, intheir entirety for all purposes. Aspects of the embodiments can bemodified, if necessary to employ concepts of the various patents,applications and publications to provide yet further embodiments.

Although the present disclosure has been described with reference to anumber of illustrative embodiments, it should be understood thatnumerous other modifications and embodiments can be devised by thoseskilled in the art that will fall within the spirit and scope of theprinciples of this disclosure. More particularly, reasonable variationsand modifications are possible in the component parts and/orarrangements of the subject combination arrangement within the scope ofthe foregoing disclosure, the drawings, and the appended claims withoutdeparting from the spirit of the disclosure. In addition to variationsand modifications in the component parts and/or arrangements,alternative uses will also be apparent to those skilled in the art.

1. A method comprising: accessing a plurality of images derived frombiological samples of patients in a cohort population, the biologicalsamples having at least one stain; computing, based on the plurality ofimages, one or more image features; deriving a diagnostic feature metricbased on the computed image features; and determining a cut point valueby applying a statistical minimization method using the deriveddiagnostic feature metric and patient outcome data from the cohortpopulation, wherein the cut point value identifies a patient in thecohort population as positive or negative for a diagnostic test.
 2. Themethod of claim 1, wherein the diagnostic feature metric is anexpression score that is selected from the group consisting of anH-score, and Allred score, percent positivity, and animmunohistochemistry combination score.
 3. The method of claim 1,wherein the diagnostic feature metric is a weighted combination ofmultiple image features or expression scores.
 4. The method of claim 5,wherein the pre-determined multiple image features or expression scoresare pre-determined are combined using a multivariate Cox model.
 5. Themethod of claim 1, wherein the diagnostic feature metric is derived froma plurality of prognostic features, wherein the plurality of prognosticfeatures are generated using a trained classifier.
 6. The method ofclaim 7, wherein the plurality of prognostic features are combined usinga multivariate Cox model for stratifying the cohort population.
 7. Themethod of claim 1, wherein the statistical minimization method is a logrank statistic minimization.
 8. The method of claim 1, wherein thepatient outcome data is a primary end point data.
 9. The method of claim10, wherein the primary end point data is overall survival.
 10. Themethod of claim 1, further comprising stratifying the patients intodiagnostic positive and diagnostic negative groups based on thedetermined cut point value.
 11. The method of claim 12, furthercomprising generating Kaplan-Meier response curves of the diagnosticpositive and diagnostic negative groups, wherein the Kaplan-Meierresponse curves indicate a percentage of patients surviving over aperiod of time.
 12. The method of claim 13, further comprisingcalculating hazard ratios based on a slope identified from the generatedresponse curves.
 13. A system comprising: one or more processors; and anon-transitory computer-readable memory storing instructions which, whenexecuted by the one or more processors, cause the one or more processorsto: access a plurality of images derived from biological samples ofpatients in a cohort population, the biological samples having at leastone stain; compute, based on the plurality of image, a plurality ofimage features; derive a diagnostic feature metric from the plurality ofcomputed image features, wherein the plurality of computed imagefeatures are combined using a multivariate Cox model; and apply astatistical minimization to derive a cut point value that identifies apatient in the cohort population as positive or negative for adiagnostic test, the statistical minimization utilizing the deriveddiagnostic feature metric and patient outcome data.
 14. The system ofclaim 16, wherein the plurality of computed image features to becombined are pre-determined.
 15. The system of claim 16, wherein theplurality of computed image features to be combined are determined usinga classifier trained to identify one or more attributes that are likelyto predict a patient response.
 16. The system of claim 16, wherein thecohort population is a placebo population.
 17. The system of claim 16,wherein the statistical minimization method is a log rank statisticminimization.
 18. A non-transitory computer-readable storage mediumencoded with instructions executable by a processor of a computingsystem to cause the computing system to: access a plurality of imagesderived from biological samples of patients in a cohort population, thebiological samples having at least one stain; compute, based on theplurality of images, a plurality of image features; derive a diagnosticfeature metric from the plurality of computed image features, whereinthe plurality of computed image features are combined using amultivariate Cox model for stratifying the cohort population; apply alog rank statistical minimization to derive a cut point value, the logrank statistical minimization utilizing the derived diagnostic featuremetric and patient outcome data; and stratify the patients in the cohortpopulation into diagnostic positive and diagnostic negative groups basedon the derived cut point value.
 19. The non-transitory computer-readablestorage medium of claim 21, further comprising instructions forgenerating Kaplan-Meier response curves.
 20. The non-transitorycomputer-readable storage medium of claim 22, further comprisinginstructions for calculating hazard ratios based on the generatedresponse curves.